Feature request: Support P64 image format

[rmueller83]

Since converting C64 .scp dumps to emulator-supported formats (.g64 or .d64) requires other tools and does not work very well, it would be nice to directly support a format which the most common C64 emulator supports. P64 files are described in the manual of the VICE emulator as "lowlevel NRZI flux pulse disk image files". The specification is here: https://vice-emu.sourceforge.io/vice_16.html#SEC300

Reference implementations can be found at https://github.com/BeRo1985/micro64disktool/blob/master/src/DiskImageP64.pas and https://sourceforge.net/p/vice-emu/code/HEAD/tree/trunk/vice/src/diskimage/fsimage-p64.c https://sourceforge.net/p/vice-emu/code/HEAD/tree/trunk/vice/src/lib/p64/

[keirf]

Which is actually more popular / used in practice? G64 or P64?

[rmueller83]

G64 is supported by more emulators and also used in TOSEC. The spec is here: http://ist.uwaterloo.ca/~schepers/formats/G64.TXT

[markusC64]

P64 is rarely used. Mainly because there was no software to create it from Fluxengine, Greaseweazle, Kryoflux, SuperCard Pro. Yesterday I released g64conv 3.0 that creates P64 from Kryoflux Dumps. And today I pushed an Alpha to github on a branch "greasewazle' that does the same for greaseweazle. Rarely tested because I do not have one. I just got a few .scp images created with it. Seems to work, but manual setting of the "d"-Parameter might be needed. That alpha also allows conversion to g64 by converting it to txt and then to g64.

While doing that, I noticed that the scp files are buggy - so the mentioned alpha only works for greaseweazle because I had to implement some workaround that does not work with regular scp files.

[keirf]

There is confusion over how to do single-sided SCP files: Whether the track numbers are cylinder numbers or cylinder+head numbers. Apparently it is supposed to be the latter (but Greaseweazle supports the other way with a command-line flag).

I may add ability to generate Kryoflux dumps.

[markusC64]

The problem is sth else. There is the "TRK" Header. Followed (but not directly followed) by Rotation Infos (Track Size, Offset of Track Data and Index Position).

Some rotations point to an empty data area.

And rotation data is overlapping between rotations, the 5 rotations that it should contain are in reality less.

g64conv is using a textual representation for p64 files (they need to be converted by p64conv to p64 because perl is to slow for p64 compression).

You might think about reading and writing that textual representation - using float instead of integers in order not to loose precision while converting clock.

BTW: I found out be examples that I need one rotation plus a few extra bytes, because rotation information from hardware is not reliable enough... for both the currently supported systems (Kryoflux & Greasezeazle).

My workaround curently is that I use the minimium start of a rotation, but not before 16k. And the maximum end of rotation. In that area I extract a whole rotation.

[markusC64]

I might add: My 1st attempt was to use the 1st rotation, append the 2nd rotation, ... and parse the resulting flux stream... Even if the point of switching rotations is wrong, that should give a correct data stream.

[keirf]

Empty TDH entries? I don't think Greaseweazle does that, but FluxEngine does for empty tracks. GW instead writes a zero entry in the TLUT at the start of the image file. GW produces continuous index-to-index dumps, for the specified number of revolutions (on command line at dump time).

[keirf]

However if there are questionable GW SCP dumps, you can attach them to this ticket for a quick look.

[markusC64]

Okay. Perhaps I should ask the one who did the dump to check his setting. Just to be sure.

No, the entry is not empty, it's "pointing" to lots of 0x00 bytes - where the rotation starting at offset 16384 clearly shows that the track contaions valid data.

[keirf]

Ok that and overlapping revolutions is certainly weird and wrong. Happy to have a look.

[rmueller83]

Here is another dump of the original C64 Demodiskette dumped with a D509V3 drive:

gw read --double-step --revs=5 --ecyl=39 --single-sided demodiskette.scp
** Greaseweazle v0.21 [F1], Host Tools v0.21
Reading Track 39.0...

demodiskette.zip

Processing this with

fluxengine convert scptoflux demodiskette.scp demodiskette.flux
fluxengine read c64 -s demodiskette.flux:t=0-79:s=0 -o demodiskette.d64

it works in WinVICE.

[markusC64]

That's interesting. For track 0 I get the following output - if i insert the debug output that is inside my code as comment: Track 0

DEBUG: 16384..69164 DEBUG: 27854..80634 DEBUG: 39068..91848 DEBUG: 50282..103062 DEBUG: 61496..114276

That are the offsets of the revolutions from the track header - the end offset is calculated by the start offset + legth (2 times, because int16 values are counted).

And for track 22 (i. e. 1541 Track 12):

Track 22 DEBUG: 16384..87600 DEBUG: 28694..99910 DEBUG: 41004..112220 DEBUG: 53314..124530 DEBUG: 89..71305

that doecodes wrong with starting form 89. With the workaround not to start before 16384 it decodes well.

Tested with g64conv.pl demodiskette.scp demodiskette.txt 1 sstd,ad1

and removed or enabled the line $minStart = 16384 if $minStart < 16384;

[markusC64]

Please forget it, I have found a typo in getting the offset... Will try again.

[markusC64]

Just pushed new alpha of g64conv. It should convert any scp file to g64 and to p64 (textual).

Still alpha because selection of the side is not possible - it is using side 0 everytimes.

[keirf]

@rmueller83 Thanks it is useful to have some known good raw dumps for testing my own C64 routines (when I do them).

[markusC64]

Yes, that is indeed a good sample. Bit I should warn that it is not representive. It is mastered so every track is aligned with the index hole. A ramdom 1541 disk will not be aligned, so the end of the rotation might fall just into the middle of the sector, which is another reason for conversion errors. For aligned disks, errors at end of rotation has no effect, unless there is a copy protection that can be broken that way.

By comparing flux around the rotation end with that of the previous rotation (which obviously does not work for the very first rotation) I get this report for the demodisk:

Reading raw track 0 Got rotation 1: 26391..52781 for track 1 Using rotation 1: 26391..52781 for track 1 Reading raw track 2 Got rotation 1: 28151..56301 for track 2 Using rotation 1: 28151..56301 for track 2 Reading raw track 4 Got rotation 1: 34365..68729 for track 3 Using rotation 1: 34365..68729 for track 3 Reading raw track 6 Got rotation 1: 34468..68935 for track 4 Using rotation 1: 34468..68934 for track 4 Reading raw track 8 Got rotation 1: 33277..66553 for track 5 Using rotation 1: 33277..66553 for track 5 Reading raw track 10 Got rotation 1: 33395..66790 for track 6 Using rotation 1: 33395..66789 for track 6 Reading raw track 12 Got rotation 1: 35289..70577 for track 7 Using rotation 1: 35289..70577 for track 7 Reading raw track 14 Got rotation 1: 33961..67921 for track 8 Using rotation 1: 33961..67921 for track 8 Reading raw track 16 Got rotation 1: 34863..69725 for track 9 Using rotation 1: 34863..69725 for track 9 Reading raw track 18 Got rotation 1: 35046..70092 for track 10 Using rotation 1: 35046..70092 for track 10 Reading raw track 20 Got rotation 1: 35325..70649 for track 11 Using rotation 1: 35325..70649 for track 11 Reading raw track 22 Got rotation 1: 35609..71217 for track 12 Using rotation 1: 35609..71217 for track 12 Reading raw track 24 Got rotation 1: 34659..69317 for track 13 Using rotation 1: 34659..69317 for track 13 Reading raw track 26 Got rotation 1: 35285..70569 for track 14 Using rotation 1: 35285..70569 for track 14 Reading raw track 28 Got rotation 1: 34829..69657 for track 15 Using rotation 1: 34829..69657 for track 15 Reading raw track 30 Got rotation 1: 35171..70341 for track 16 Using rotation 1: 35171..70341 for track 16 Reading raw track 32 Got rotation 1: 35433..70865 for track 17 Using rotation 1: 35433..70865 for track 17 Reading raw track 34 Got rotation 1: 24907..49813 for track 18 Using rotation 1: 24907..49813 for track 18 Reading raw track 36 Got rotation 1: 32399..64797 for track 19 Using rotation 1: 32399..64797 for track 19 Reading raw track 38 Got rotation 1: 31865..63729 for track 20 Using rotation 1: 31865..63729 for track 20 Reading raw track 40 Got rotation 1: 31665..63329 for track 21 Using rotation 1: 31665..63329 for track 21 Reading raw track 42 Got rotation 1: 32845..65687 for track 22 Using rotation 1: 32845..65689 for track 22 Reading raw track 44 Got rotation 1: 30975..61949 for track 23 Using rotation 1: 30975..61949 for track 23 Reading raw track 46 Got rotation 1: 31821..63641 for track 24 Using rotation 1: 31821..63641 for track 24 Reading raw track 48 Got rotation 1: 22691..45380 for track 25 Using rotation 1: 22691..45381 for track 25 Reading raw track 50 Got rotation 1: 22261..44520 for track 26 Using rotation 1: 22261..44521 for track 26 Reading raw track 52 Got rotation 1: 22267..44533 for track 27 Using rotation 1: 22267..44533 for track 27 Reading raw track 54 Got rotation 1: 28455..56909 for track 28 Using rotation 1: 28455..56909 for track 28 Reading raw track 56 Got rotation 1: 30993..61985 for track 29 Using rotation 1: 30993..61985 for track 29 Reading raw track 58 Got rotation 1: 30197..60393 for track 30 Using rotation 1: 30197..60393 for track 30 Reading raw track 60 Got rotation 1: 25801..51601 for track 31 Using rotation 1: 25801..51601 for track 31 Reading raw track 62 Got rotation 1: 27447..54893 for track 32 Using rotation 1: 27447..54893 for track 32 Reading raw track 64 Got rotation 1: 28471..56941 for track 33 Using rotation 1: 28471..56941 for track 33 Reading raw track 66 Got rotation 1: 28383..56764 for track 34 Using rotation 1: 28383..56765 for track 34 Reading raw track 68 Got rotation 1: 28993..57985 for track 35 Using rotation 1: 28993..57985 for track 35

You see there are many tracks where the rotation length is corrected a little bit. The same effect will break unaligned disks when that correction of the rotation length is not done. Do not ask why, but this effect has been observed on many disks - and I can see it in the .scp images I have for testing.

[keirf]

The dump has five consecutive rotations. There should be no need to cut one out and then fudge around splicing its ends together.

[markusC64]

For sure. But the rotations are not exact - you see, the The end of the rotation was detected with a few flux difference to the real end. So you have either drop a few flux or append a few from the next rotation. For that, you need to compare a range of ± 250 flux (or another range, I had problems with a range that was to small because of a repetive pattern in the flux) the beginning of the rotation with its end to find the best offset for adjustment.

[keirf]

Oh yes, hang on, I suppose indeed if you are doing minimal conversion SCP to P64 then you need to do something like this. My primary interest is in fully analysing raw dumps and regenerating track framing for destination image file. If you want a more straightforward conversion then yes you pretty directly have to find the track write splice point, and/or stitch the track start/end together just right. Agreed :)

[keirf]

Put another way: I will not be doing raw dump or raw conversion to P64 (or a very wide range of other single-rev formats). Perfectly understandable there are other tools out there filling a desire to do just that, though.

[markusC64]

You wrote: "My primary interest is in fully analysing raw dumps and regenerating track framing for destination image file."

While it is reasonable to do so, it will not work in all cases. E. g. Geos 1.2 has written raw bytes to 1541 track 36 without any sync and any frame. That are checked for the purpose of copy protection. There are even commercial c64 disks that uses a custom sector format that has nothing to do with the standard format - even not having syncs. Just a few sectors use standard format so the loader can be loaded. You will not be able to convert them to g64/p64 by regeneratig track framing because you do not know what to look for. You can only do the very same decoding a 1541 would do. And for that you need the corrected single rotation.

But anyway, it's perfect when both software are extending each other. I do not see a problem with that. Just wanted to let you know that POV.

[keirf]

Yes it is a lot of work. I have not finished it on Amiga for example, but see the approach is to define format specifiers for all titles: https://github.com/keirf/Disk-Utilities/blob/master/disk-analyse/formats

[vintagepc]

In the interest of sharing knowledge, while searching for how to write D64 files with my GW today, I stumbled across this tool, which does just that out of the box using the GW

https://csdb.dk/release/index.php?id=218378

It works as advertised and was able to write an alignment program for me so I could run it to sanity-check a second 1541 on my C64.

Alas, no visible sources one could leverage to make it a mainline feature, and it's winderp only