Writing kryoflux-stream of C64 disk problem

[ILAHWWINC]

When writing a scp file from a kryoflux-stream of a C64 disk the result is a only partial working disk. The author of g64conv reports that there is a problem with writing at the end of the rotation. There is a special alignment nessecary to make the writen disk working.

I attach 2 files to compare. Any hints for solving this Problem?

P.S. I hope the author of g64conv will add technical details to give a better explanation.

g64conv.zip

[markusC64]

The original image is a kryoflux stream. While conerting to scp, halftracks were dropped, so we have an image equivalent to beig read with double step.

1541 images are generally not aligned to the index hole. It happens often that the index hole is inside a sector.

When writing tracks, at least one bitcell is broken almost everytime.

From my POV, the only solution is to find a position where that error has no negative effects and use that position to stop writing.

Fot that, extending the scp format to insert hints where to stop writing might be an idea. You see, in scp format, all is index based. I think we can easily insert a block after the position table of the tracks with marker bytes containing the hints. All other software wil ignore them because all is index based without absolute positions.

[markusC64]

Oh, one remark I forgot to mention: The scp generated from g64conv contain 3 revolutions, each exactly the same and are exactly one rotation, not even a part of a bitcell more or less. Suitable for one rotation formats.

[keirf]

There are a few options. One is to include metadata as you say. Another would be to rotate tracks to place a safe spot at the index. Another would be to generate a "mastering" SCP with a revolution-and-a-fraction of data: That is, the second revolution is incomplete and ends where we would like the track splice to sit. This is essentially how I write non-aligned tracks anyway (when I know they are non-aligned, and I know where to put the splice).

[markusC64]

One is to include metadata as you say.

This avoids the disadvantages I mentioned below.

Another would be to rotate tracks to place a safe spot at the index.

Does not work for some copy protections. Currently available as workaround in g64conv 3.1 rc from git master as an extra processing step the user must start.

Another would be to generate a "mastering" SCP with a revolution-and-a-fraction of data: That is, the second revolution is incomplete and ends where we would like the track splice to sit.

While this is possible, they are uncompatible to some scp software. Including g64conv - that's actually the reason why I added 3 revilutions to the scp: Alowing comparing to revolution before and next the actual one in order to extract exactly one revolution - tools do not know that this is already done.

This is essentially how I write non-aligned tracks anyway (when I know they are non-aligned, and I know where to put the splice).

If I understand correctly, you're telling us that this should work right now?

[markusC64]

E. g. the Supercard Software is said (by its documentation) to have auto splicing for c64 images. That means we should not break that by having partial revolutions stored.

[keirf]

It can work any of these ways very easily as far as I'm concerned. It won't work for SCP images right now, but if you define an extension how you like, I can detect and support it. Personally I'm happy with option (3) too but I'd be viewing the SCP as a disposable "carrier format" just as part of a write pipeline between your software and mine: G64/P64/whatever would be the "gold image". But I don't care enough to have a philosophical debate about it. If you hack it up, I'll support it.

[markusC64]

I can understand your point. But please view it from the other side: Sides like c64 preservation project, wiki.retronn.de and likewise will get those special purpose scp from users and this way receive scps of limited quality. I'd like to avoid that.

I'll specify & implement that extension in a later version of g64conv. g64conv 3.1 is planed to be released this weekend without any further modification unless there is a showstopper bug found.

Edit: In contrast, flux data with no variation can be detected easyly. And three equal revolutions can be detected easyly, too. And flux data just converted from another flux format are not bad.

[markusC64]

but if you define an extension how you like, I can detect and support it. [....] If you hack it up, I'll support it.

I have implemented such an extension in g64conv a few minutes ago. Let me describe it a bit:

At Position 0x2b0 in the scp file, there is an additional header with magic Bytes "WSP" followed by flags, where up to now only bit 0 is defined and the other buts are reserved and should be 0. Bit 0 tells me if the revolutions are ideal rotations so the algorithm matching start and end of rotation need not to be applied for extracting a single rotation format. For g64conv generated scp files, it should be set.

After that flag byte, a 32 bit integer value follows that is time in 25ns clocks describing how much of the last rotation should be written. I have chosen time because that allows to place the end of the writing area in a flux free area if such exists. Remember scp images are generated for some rotation speed, it might be 360 rpm, but it can be e. g. 360.71 rpm. The "write splice times" are for only for that rotation speed. If the drive's rotation speed differs, you'll have to do an arithmetic correction.

You might not know, but using the original supercard pro soft- and hardware, you have to create the scp file with exactly the rotation speed of your drive. That's the reason g64conv can create scp files for different rotation speeds. So if gw could output the current rotation speed in units of 25 ns clocks, that would be fine for creating such scp files. If gw would do an arithmetic correction for different rotation speeds, that would be best, but knowing the rotation speed in units of 25 ns might be fine, too.

A current limitation to the beta of g64conv is that write splice hints are only transfered from g64 to scp, they are not created. So you need a g64 with write splice information - which is possible via kryoflux g64 mastering extension of g64.

The attached g64 is such an example.

geos20d1WithWriteSPliceInfo.zip

g64conv.pl 1.g64 sample.scp 6666666 will create an sample.scp for you. The integer is the time per rotation in 25 ns clocks. I already executed that:

sample.zip

Up to now, I have no idea to verify the write splice hints in the scp file... They should be ok, but I haven't verified.

[markusC64]

Oh I forgot to mention: write splice time "0" means: Do the standard writing method, no special treatment is necessary.

[keirf]

Okay so that's an array of 32-bit little-endian write-splice time values. That's fine, no problem at all.

I would like to suggest we define this extension area a bit more extensibly. In case we want to crowbar in extra things beyond WSP. So for example a chunk-based format something like:

4 bytes: EXTS 4 bytes: Total size of Extension area

Extension area is a list of chunks: 4 bytes: ID 4 bytes: Size of chunk Size bytes: chunk-specific stuff

Which for you could be: 4 bytes: WRSP 4 bytes: size (ntracks4 + 4) 4 bytes: flags/version ntracks 4 bytes: splice times

What do you think?

[markusC64]

In general, I'm fine with that. But I do not get the benefit of the extra "EXTS" header. You see, Extensions are from 0x2b0 to the very first TRK Header... which means when you hop from one extension to the next, the very first TRK Header will tell you when you're done.

Anyway, in "sub txt2scp" is the conversion in my perl file - I (or you) can adjust the header to get a test file...

May I ask how gw works with different rotation speeds? Original Supercard Pro hardware does not really, as I wrote above. You see, that's important for writing a howto (when I find time to write that, I know...).

BTW: I have seen you support kryoflux streams directly. Fine, but often writing them back with kryoflux does not work either - you see, they do not have write splice information and kryoflux does no write splicing for writing stream files. Might be quite hard to make it better than kryoflux. But for many disk types not doing splicing should work.

[keirf]

The idea is to give a simple single way to detect this extension format. What if the first chunk were one your parser does not recognise? Do you just push onward and assume it conforms? It is better to have a single first enclosing header, which everyone matches on, and then can safely assume all further chunks conform even if some IDs are not understood.

Greaseweazle measures rotation speed before beginning the write process, and adjusts all tracks to the measured rotation speed. So actually the rotation speed you provide in SCP does not matter, as it will get auto adjusted.

I have just added support for Kryoflux streams. It is mainly for interoperability, and also I will provide format conversion in future (eg. input Kryoflux, analyse as IBM track format, output to an IMG file), so in that case taking KF stream as input will indeed make more sense.

[markusC64]

The idea is to give a simple single way to detect this extension format. What if the first chunk were one your parser does not recognise?

That's a valid point.

Greaseweazle measures rotation speed before beginning the write process, and adjusts all tracks to the measured rotation speed. So actually the rotation speed you provide in SCP does not matter, as it will get auto adjusted.

That's fine - then users can ommit that argument - it defaults to 6666666, which is 360.000 RPM rounded to integer (you see, the counter increment would not have happened, so rounding down is right).

I have just added support for Kryoflux streams. It is mainly for interoperability, and also I will provide format conversion in future (eg. input Kryoflux, analyse as IBM track format, output to an IMG file), so in that case taking KF stream as input will indeed make more sense.

Perhaps it's time to look for a CPM Disk (MFM) in kryoflux format... mostly IBM format, you see. A CPM Disk is missing in my image collection.

[markusC64]

Just changed extension format in my git master branch of g64conv. sample (2).zip

[keirf]

In your sample file, the EXTS length field is 4 bytes short isn't it? The WRSP chunk looks good.

[markusC64]

Thanks for checking. I must have miscalculated the length - you see, it's hardcoded in the patch...

[keirf]

I did see :)

[keirf]

I will add support for this to GW and report back.

[markusC64]

Ok. The mentioned g64conv command is actually a shortcut for

g64conv.pl filename.g64 tmpfile.txt p64 g64conv.pl tmpfile.txt filename.scp [timePerRotation]

and that textual flux representation in the tmpfile.txt has been checked by decoding back to g64 - that is correct. This info might help if some debugging is needed.

With current g64conv.pl from git master, write splice info can be added to any g64 by converting it to text and inserting "write-splice-position 0" at the place where the write splice position should be set to. And then converting back the text to g64. I know some automatic heuristic for that is needed, but that needs time to develop.

[keirf]

The support is now added to Greaseweazle, see above. Unless there is further testing to try, I will close this ticket.

If you would like me to do a write-then-read test let me know, and what runes are needed to create the SCP for write, and to analyse the SCP for validity after (if that is possible?).

[markusC64]

Yes, testing is necessary.

@ILAHWWINC Can you help testing?

@keirf The Image I gave you should have no read errors in a real 1541/70/71. And it should boot geos with the german message to insert the backup system disk.

[keirf]

I don't have a C64 with 1541 set up. Is it cbmdos? As then I could possibly read back and process to D64 using fluxengine.

[markusC64]

It is - with copy protection in the header and tail gaps. Fluxengine will give you a 1st impression if it worked, but not a final decision.

Not strictly related to this issue, but related: For a 1541 to be able to read the disk, it has to be wiped before. This can be done by using a magnet. Or by the drive itself.

Note: Wiping disks with kryoflux gives very bad results. Doing the same with original supercard pro hardware gives very good results using the same drive.

But you can use my perl skript

g64conv readback.scp readback.g64 stddisk,ad1

to convert it to g64 and test it with VICE, if you like.

[markusC64]

g64conv readback.scp readback.g64 stddisk,ad1

Or even better:

g64conv.pl readback.scp readback.g64 stddisk,ad1,d10000

[keirf]

I tried conversion but got:

viper:g64conv$ ./g64conv.pl ~/Greaseweazle/a.scp a.g64 stddisk,ad1,d10000
UNKNOWN stddisk
Died at ./g64conv.pl line 3920.
viper:g64conv$ ./g64conv.pl ~/Greaseweazle/3.scp a.g64 stddisk,ad1,d10000
UNKNOWN stddisk
Died at ./g64conv.pl line 3920.

Note that 3.scp above is the original converted SCP image file that you attached earlier. So I can't do successful conversion to g64 even with that?

EDIT: And I get the same error if I convert the original 1.g64 to scp myself, using g64conv, and then try to convert the SCP back to G64.

[markusC64]

Oh shit, I should know my arguments for g64conv :-(

This workes for me:

g64conv.pl 3.scp 6.g64 sstd,ad1,d10000

It's "sstd", not "stddisk"...

"ad1" seĺects the faster flux decoder, wjhich should be sufficient for this case. "sstd" uses standard speedzones instead of detection - so this cannot go wrong. "d10000" setzs a larger allowed range for rotatioon correction. For scp, this is often necessary. No idea about the exact value needed. 10000 is large enough and probably larger than needed.

[keirf]

Ok, 3.scp converted to g64 does work in VICE. If I write it and then read it back, convert to g64, it doesn't work in VICE.

So something is wrong somewhere.... either at your end or at mine. I will have to take a little look later... Perhaps it will be obvious from flux pattern if the write splice is in the track gap as it should be, if I do a flux plot?

[keirf]

A quick look at the dumped flux tells me the drive is a bit unhappy with the flux timings (I'm using a PC 3.5 drive). Maybe I'll try a more permissive 3.5 drive. Unfortunately modern PC drives hate non-MFM timings.

[markusC64]

You can also try

g64conv.pl 3.g64 6.txt 1 sstd,ad3,d10000

(for this extra feature, you need the slower decoder, it will show you the write splice - search for "write-splice" in the resulting file.

You can do the same with an scp file, but you will not get the write splice.

In the mean time I'll test the very same scp file with original scp hardware... Just to be sure. Of cause it will not respect the write splice hints, but it will do its own write splice position...

[keirf]

A more permissive drive (an old Amiga Teac drive) does much better. I can list the disk contents, and load GEOS, but then it returns me to the BASIC prompt rather than getting to the requester in German.

[markusC64]

I can list the disk contents, and load GEOS, but then it returns me to the BASIC prompt

Clearly the copy protection is broken. It's "55 67" in the gaps in contrast to standard "55 55 ... 55". Can be a wrong generated write splice by dtc or a problem...

On track 21 there are 19 sectors, which is the Commodore DOS standard. The copy protection check checks all 19 header gaps of track 21. The copy protection check checks the first 18 sector data gaps of track 21. The gap between the last sector and sector 0 is not checked.

[keirf]

Ok, I will dig into this a bit more later. At least the flux looks like it's banded correctly now with the Teac drive, so as you say I think this must be a splice problem, in all likelihood.

Here is my SCP image file dump: https://drive.google.com/file/d/1CgQACA2lJwIIRXltx5rBdmxATZDvUWrE/view?usp=sharing

[markusC64]

Thanks. I'll examine that file.

[markusC64]

Lets's have a look.

Well, interesting enough, your file works for me using my private WinVICE 3.2 Build.

` gcr 00 gcr 00 bytes 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 54 d4 0a aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ab 55 55 ff

sync 32 ; header gcr 08 begin-checksum checksum 13 ; sector gcr 00 ; track gcr 15 ; id2 gcr 4a ; id1 gcr 4c end-checksum gcr 0f gcr 0f ; Trk 21 Sec 0 bytes 55 55 55 55 55 67 55 55 67 bits 111111 `

You see clearly in the last data gap where the write splice has been - it's exactly the position where the "55 55 55" pattern is broken. As explained, that area is not checked.

I think I'll locate the write gaps in all tracks to check...

[keirf]

I used Ubuntu packaged VICE, and the original 3.scp converted to 64 works for me.

By the way, I note that your 3.scp has 360rpm track timings (166.67ms per revolution). I thought 1541 spins at 300rpm?

My dump spins at 300rpm as you'd expect.

[markusC64]

I do not know why but the write splice position seems to be about 2 flux to early in Track 21.

The flux data in the scp is - you know - delta time between previous flux and the actual one. Does that say anything where the very first flux on the track is located? That might be the difference that is to be explained.

[markusC64]

By the way, I note that your 3.scp has 360rpm track timings (166.67ms per revolution). I thought 1541 spins at 300rpm?

I know. Made because when writing back with PC drive, rounding errors are minimized as PC drives spin at 360 RPM. Edit: To be clear: PC HD Drives 5,25″.

[keirf]

Two flux too early? Is that like, what, 10us?? I think mechanical variation can explain that!

EDIT: I mean, 10us in a minimum 200 millisecond write is an error of 50ppm. Overall that error is 25ppm < E < 50ppm. Isn't that actually awesome? ;)

[markusC64]

I see a few bits changed that shoould be written exactly before the write splice (and therefiore should be correct), counting only "1" gives two flux.

You see, I'm checking the write splice position by comparing the data written with the bytes read. And in the textiual g64 representation the write splice position is marked.

[keirf]

Can you give the expected and actual data in the region of the splice?

[markusC64]

I don't know what exactly you're asking for.

g64conv.pl 1.g64 expectedData.txt

g64conv.pl a.scp actualData.txt 1 sstd,ad1,d10000

shoudl give you both. Search e. g. for "trk 21 sec 0" case insensitive in both to find the area which I checked - a few lines above the location you'll find.

The first will contain a line write-splice-position -4 which means that the write splice position is 4 bits before the position where you have found that line.

Edit 1: geos20d1WithWriteSPliceInfo.zip contains "1.g64" Edit 2: It's possible that my write splice time is not perfect - you see i noted a few postings before: "The flux data in the scp is - you know - delta time between previous flux and the actual one. Does that say anything where the very first flux on the track is located? That might be the difference that is to be explained."

[keirf]

It looks perfect enough to me. Expected is 55 55 54 0, aa aa aa. Dumped is 55 55 54 d, 40 aa aa.

So we have twelve cells of splice in the dump. But looks in the correct place at least (that 0 in the expected is a 4-bit artificial splice I guess). Nothing is actually checking this splice, right?

EDIT: On another write attempt I've dumped 55 55 54 a, 85 55 55. Which is a bit better. There is some luck involved.

[markusC64]

Yes, for this g64 image this is not checked... In my understanding, the bits before the write-splice-position should be exact, then the write splice position comes with a few bits wrong (insertded, deleted or changed) and the rest should be correct again.

I do not understand why the "0" is changed in "d", that should be before the write splice position. But that's a matter where the very first flux is located on the track, I suppose. A hex digit zero indicates that there are about 14 µs no flux. Imagine that would be the beginning of a rotation, not in the middle. Part of the 14 µs could be at the end of the rotation (it's a circle)... So where is the first flux? The difference between all possible interpretations are 14 µs - ε. All times between between neighbour flux remain the same. Perhaps Interpretation when creating scp and writing differs.

I think such an effect also makes the difference why you have read "d" instead of "0"...

[keirf]

The reason is that you can't actually write out that final length of flux and have it read back consistently. The end of your track is 540, which is 6 zero bits at the very end. Each zero is 3.5us at 300rpm, so that is 6*3.5=21us. Most drives start reading random crap if they get no flux transition after 10us or so. Essentially they crank up their automatic gain control on the read amplifier and start reading randomness.

[keirf]

I am also, as it happens, not properly writing out the very last flux. But fixing that made very little difference, because your final flux is invalid as described above, so I still read that 0 as d sometimes. I will however look at tightening up that end-of-write condition. But I think largely we're dancing around something here that doesn't matter very much.

So the dump does actually work for you fine, and apart from this quibble over track 21, seems okay?

[keirf]

My recommendation actually is to end the write before the 0 nibble in track 21. That 0 represents the splice point where the erase head has erased ahead of the write head (try saying that 10 times fast!!) but the write head stops before getting to the 0 nibble. Does that make sense? So then last flux would be more like 1 or 2 bitcells size. Perhaps I should try mocking that up.

[markusC64]

I agree, for this copy protection it does not matter.

But there are starnge copy protections that use only half of the track - for one halftrack the 1st half, for the other the 2nd. For such cases, understanding is necessary...

Anyway, current implementation seems to be quite good - on both gw and g64conv side.

BTW: This has been an artifical image. It has been copied with a 1571 before in order to ensure that splicing is really needed. And it has been chosen because copy protection checks nearly the wohle track 21 - except the very last gap. Gaps are checked by copy protection and blocks are read normally - in the sum everything is checked. So you can be sure that the write splice is quite good if the image works.

[keirf]

Yes isn't there one title only that spirals out in half tracks? I assume switch happens in a sector gap, so some small discontinuity / timing mismatch between half tracks is allowed? Thinking aloud, you'd want the disk wiped, and then you'd want to write a track but step in the middle? I'm not sure all drives allow that but maybe old drives do. Or worst case stop the write, step, and precisely time the start of the write? Haha, I'm not sure... Some careful experimentation would be required and it would I'm sure be a very customised mastering process! :)

[keirf]

By the way in all my mucking with the end of the write, I've made very little consistent effect on the actual write splice. It really is messy for approx 20us. With my drive. I think this can vary :)