Reading Z64K formatted Big Blue Reader MS-DOS .g71 fails on U64

[Jusalak]

I experimented a bit with Big Blue Reader. I used Z64K C128 emulator and Big Blue Reader to format a MS-DOS 360kB (MFM) disk image to .g71 file. Then I copied a text file from a PC to a GCR disk using DirMaster (with ASCII > PETSCII translation). Then I used BBR to copy the file from the GCR disk to the formatted MS-DOS disk (with back-conversion PETSCII > ASCII). Then I verified the file using BBR C128 and C64 versions, and success, Z64K with BBR (64 and 128 version) access the file on MS-DOS disk image perfectly fine.

Then I tried U64 and BBR 64 (with virtual 1571), but loading the disk failed. I wonder what is the cause, the .g71 image itself seems valid since Z64K reads it without problems.

Files attached (original text file, GCR disk image and the final MS-DOS formatted disk image).

link to BBR 64: https://www.planetemu.net/rom/commodore-c64-applications-d64/big-blue-reader-64-1988-sogwap-software link to BBR 128: https://commodore.software/downloads/download/161-data-conversion-tools/1165-big-blue-reader-128-v4-10

MSDOS.zip

[GideonZ]

Interesting.. because the .g71 file format was actually defined while doing mixed format (GCR/MFM) disk support for the Ultimate. :-) Is it possible that the definition of this .g71 file format diverted, as it was defined elsewhere in a different way?

Message ID: @.***>

[Jusalak]

I don't know the particulars, there apparently are some differences in implementation. I can't do the converse, since BBR 64 does not support formatting MS-DOS floppies (only BBR 128 does). The author of Z64K is a friendly person and I think he will help to resolve the issue, if needed.

Edit: I noticed that the BBR 64 disk has a MS-DOS formatting utility after all. After a quick test it appears that Z64K can read U64 generated MS-DOS .g71 files normally.

[Jusalak]

Z64K recognizes the empty MS-DOS image generated by BBR 64 MS-DOS formatting tool on U64, but not the other way around. Both images have 362 496 bytes free, a familiar number to PC users. emptymsdos_g71.zip

[markusC64]

That's interesting. My g64conv can decode "U64emptymsdos.g71" well, but takes quite long for "Z64Kemptymsdos.g71". Will have to wait - but I think I have an idea what is going on here.

[markusC64]

Oh well, a strange format. I have identified it.

Basically, the difference is that the U64 uses a format that is mostly a sector image - comparable to td0 and imd formats. Z64k has generated a format at bitstream level (much like what is g64 for gcr). But g64conv has decoding problems with it, because I have implemented a similar extension to G64 differently (and better, because my extension supports more: Track length does not need to be a multiple of 8 bits and support for longer tracks observed at HD and ED MFM disks). If you understand MFM a little, you know the bitstream consists of alternating clock and data bits. These bits were packaged into bytes (8 bits / bye, that is 4 clock and 4 data bits) and saved to the g71 image.

If I have analyzes the format of the example g71 correctly, the format Z64k has used does not support arbitrary track length (which is bad, when you read disks with kryoflux / fluxengine / greaseweazle / ... the track length are usually not a multiple of 8 bits. For the 1571, HD and ED are neglectable, I agree.

The author of Z64K is a friendly person and I think he will help to resolve the issue, if needed.

Can you direct him to this issue? If something evolves that is below the scope of this issue, one can switch to private communication. But in general it's of interest for the ultimate, because it might want to support different formats, too.

Edit: Indeed. I have hacked my g64conv by commenting out the part for arbitrary lengths of FM / MFM Tracks. And now the sample images decodes well.

[Jusalak]

I will notify him, but if you have questions it is probably better for you to engage in direct correspondence about the particulars. Use the contact form at: http://z64k.com/contact.php He will most likely answer, after some delay perhaps.

[markusC64]

Well, I understand. But Gideon might want to implement that format, too. Perhaps bad to explain it several times - and cross communication to get them all to the same format. And so on.

[Jusalak]

I notified him. Now it is up to him to answer, but I am sure he does.

[Jusalak]

Is g64conv available? If it is, I might want to try it as well.

[markusC64]

Of cause, it is - see https://github.com/markusC64/g64conv And see https://github.com/markusC64/g64conv/releases/tag/v4.3 for Windows binaries.

The hack for making it compatible with Z64ks format (just reading, not writing) is to patch the perl source as follows:

Remove the lines that are comments here Around line 1304: if ($trackSize > 32767 && !$isMFM) { $isMFM = 2; $trackSize -= 32768; }

HACK $trackSize *= 8 if $isMFM == 1;

  $trackSize *= 8 if $isMFM == 3;

Around line 1422:

HACK

HACK if ($isMFM == 1 || $isMFM == 3)

HACK {

HACK my $bitsToRemove = ord(substr($trackBin, -1, 1));

HACK $trackContentBin = substr($trackContentBin, 0, length($trackContentBin) - $bitsToRemove);

HACK $trackSize -= $bitsToRemove / 8;

HACK }

And be warned, g64conv is soetimes quite slow - so slow that I have decided to develop a new tool which is much faster... but the new tool does not support FM and MFM yet - and is not available, because User Interface is completely missing.

[Jusalak]

With Z64K C128 CP/M MFM-formats also seem solid, as far as I have tested them, but obviously can't test those files with U64.

And BBR manual is also available at: https://commodore.software/downloads/download/211-application-manuals/13335-big-blue-reader-64-128-v3-1-manual

[markusC64]

since BBR 64 does not support formatting MS-DOS floppies (only BBR 128 does

This is not true, Version 4.12 of BBR 64 also supports formatting disks.

[willymanilly]

I had a conversation with a user on this exact topic about a year ago. During that conversation he contacted Gideon who provided details on his custom format with g71 to support MFM from which I included reading of ultimate g71 disks for Z64K.

MarkusC64 has already worked it out. Basically, I wanted to stay aligned with the intent of G64/G71 image format. The following is a copy and paste of my explanation of Z64K's implementation for writing g71 disks with combination of GCR and MFM data.

"FYI, now that the 1571 supports MFM, any disk format a real C128 can read/write, Z64K should be able to support that now. The only way to save images with MFM formatted tracks in Z64K at this stage is with the g71 file extension. Z64K automatically detects the track type and speed zone for every track so you can actually have a combination of GCR and MFM formatted tracks. For MFMI use a nonstandard speedzone of 8 in the g64 file format standard. Without getting too technical, the reason I decided on value of 8 for the MFM speed zone because it fits nicely with this formular. 100000/(16-zonesize) = number of bytes on track. A MFM track requires 12500 bytes per track when including the clock bits. This allows any format of disk to be saved. You can even have a disk with a mixture of GCR and MFM tracks with totally flexibility on the sector count and size on any track."

For saving of MFM/GCR disk images I felt the above was sufficient to handle the majority of scenarios of disk formats the 1571 is capable of creating but am very open to improving to a better standardized format.

[GideonZ]

Okay... Thank you for replying Willy...

The Ultimate uses a binary format, and also allows any valid MFM format to be saved, even with mixed sector sizes. Since the disk can only be read through the MFM controller chip, there is no need to support bit-wise disk dumps. In fact, the whole emulation doesn't work like that; it works on a command basis. There is no MFM encoding / decoding involved.

So, in order to support the bit-wise format, the Ultimate application simply needs to decode the MFM track into sectors and put the sectors in memory. Of course, when saving, the ultimate format will be used, but I assume that the Z64K / Z128K can read this format.

Best regards, Gideon

Message ID: @.***>

[willymanilly]

Yes, it should be as simple as decoding the track into sectors.

I confirm that Z64K supports reading the ultimate format of g71 images. I do have plans for Z64K to have the option to save using the ultimate format which should be relatively simple to do. I will revisit this in the near future now that it's on my radar again.

Cheers

Will :)

[markusC64]

Well, I'll write you later this day details. I think it is important that the G71 format used can store raw (i. e. bitstream) MFM data whith a size that is not a multiple of 8 bits. You see, real rotation speeds are near 300 RPM, but not exactly 300 RPM. So the real track size differs from the expected one a little. And there is no reason why the size should be a multiple of 8 bits, so in reality isn't.

So for g64conv I also used Speedzone 8 for Double Density MFM, but with a hack that allows arbitraty track length... Now I have an appointment and thus no time to go into details. I will submit them later.

Apart from the fact mentioned I agree, that is the way to go.

[markusC64]

Well, at the time I implemented that in g64conv, I had to targets that the extended g64 / g71 has to meet:

(1) Kryoflux, GreaseWeazle reads flux from original disks which can be converted into bitstream data. This bitstream should be valid for the g64 resp. g71. Padding it to a multiple of 8 bit is bad because you don't know where to add bits without breaking a possible copy protection. For 1541 the same is more simple, if you enlarge a sync by 1 bit, the 1541 does not really notice it. Which does not mean that this is good for GCR tracks, but that's how g64 stores gcr tracks. Since MFM tracks were new, it seemed to be wise to decide better.

(2) MFM DD, MFM HD 3.5″, MFM ED 3.5″ and MFM HD 5.25″ shoule be able to be represented (3) FM DD should be able to be represented.

Okay, Point (2) were easy: Speedzone 8 is MFM DD, 9 is MFM 3.5″ HD, 10 is MFM ED 3.5″ and 11 is MFM 5.25″. 12 is FM DD. Speedzones 13-15 are free for later additions.

Point (1) is the crucial one: Since the g64/g71 extension of the ltimate for storing MFM prohibites us from setting the bit 15 of the length byte to 1 for any other format but the extension introduced by the ultimate, it was clear that we cannot store the length. So the idea was: Divide the length by 8. Then it can be stored. Is this a problem? No, just pad the data to be a multiple of 64 bits as follows: Let n be the number of bits for the track. Then calculate b = 120 - n % 64. Add b times a zero bit. After that the intermediate bitstram's size is a multiple of 8 bits. And then add p=b+8 as Byte. Done, now the data size is a multiple of 8 bytes. When reading, remember you have to read 8 times the size. Take the last byte's value. It says you how many bits you have to remove from the end.

You get that (a) Even bit rates slightly beyonf ED could be stored (b) No necessarity for bistream to be a multiple of 8 bits (c) Software not knowing this special case will just read 1/8th of the track, so one will probably notice quite fast that "there is a proble with reading the data".

@willymanilly What do you think? Is this a good extension of the g64/g71 format for MFM disks?

[willymanilly]

I note your intention seems to be able to support more than 1571 disk images. Can I suggest maybe using a different file signature for your proposed format? Both Z64K and Ultimate use "GCR-1571" as the file signature to indicate a double sided 1571 compatible disk. If you provide some example images using your proposed format I am happy to support it. Z64K currently ignores the file signature value but that can be easily updated to be able to support deviations of the g71 format.

Using the defined g64 image format at http://www.unusedino.de/ec64/technical/formats/g64.html, in summary the g71 format file header to support double side g64 that Z64K uses for creating g71 images is :-

  Bytes: $0000-0007: File signature "GCR-1571"
               0008: G71 version (presently only $00 defined)
               0009: Number of tracks in image (usually $A8, decimal 168)
          000A-000B: Size of each stored track in bytes (usually  7928, or $1EF8 in LO/HI format
                     When MFM speed zone 8 is used this value is increased to 12500, $30d4 in LO/HI format)
                     note: Z64K ignores this value, instead using other aspects of the G64 specification
                           to determine track size etc. 

As noted in my previous post, the logic of why I use speed zone 8 for MFM ut fits in nicely with the 100000/(16-zonesize) that gives the standard expected byte size oer track for the varying speed zones the 1571 can produce at exactly 300 RPM.

MFM             (8) = 12500 bytes (4 clock and 4 data bits/byte)
GCR track 1-17  (3) = 7692 bytes
GCR track 18-24 (2) = 7142 bytes
GCR track 25-30 (1) = 6666 bytes
GCR track 31+   (0) = 6250 bytes

[markusC64]

Well, I have to think about that - one the one hand, if you have a MFM disk that you formatted in a 1541, you'll get a mixed disk. That has no need to be broken with existsing emulations because tge MFM on tracks 36 to 40 isn't used at all.

One the other hand, it might be possible to heuristically detect my modification. 8 times the length must fall into the image, must not overlap with other tracks and the padding must be the correct one. And for emulators also plausible track length. For g64conv and likeweise, it's fine to be able to convert pieces of a track in order to have it changed from hex to decoded and back, so for that categorie of programs, the plausibility of the track length is a bad confition.

I am not sure (but I doubt there are existing images), so I could switch to speeds 16 to 31 for the "arbitraty lengtg" mod of speeds 0 to 15. Then it does not conflict any more. And since speeds 16 to 31 makes no sense on the hardware of a 1541/1571/... I don't see a problem as the positions 16 to 31 are still in the header and therefore cannot occur on valid multi speedzone per track images.

And yes, for CMD FD, you need all the 3.5″ "speeds". And when you have a software that does MFM decoding to textual, why not use it for 5.25″ HD, too?

PS: The VICE team intends to specify G64/71 Version 2... In order to avoid conflicts, I cannot just increase the version byte. Therefore I think for now it's best to add features to version 1 to up to now unused speed zones. And collect requirements this way.

[GideonZ]

I would vote against heuristics at any time. Please, let's apply the KISS principle.

[markusC64]

I know. And I am really unhappy. But current situation is that two different implementations of speed zone 8 are already realeased. With the possibility that images are availabe and you cannot see directly which format of the two they follow.

In the last two days I had no real idea for that... Trying to detect them might help... We really need G64 V2 - but that does not help for existing images and existing emulators.

Given that I don't have an idea that is really KISS.

[GideonZ]

I totally understand your eerie feeling, Markus. On the other hand; let's be realistic: how many images are actually out there now that have this format?

[markusC64]

Right, that's why I consider changing the software to have speedzones 16 to 31 for the arbitrary length mod, mirroring speedzones 0 to 15 with that additional feature.

An image conversion program might want to handle the special case for existing images, but emulators then don't need to.

But that only makes sense of @willymanilly also implements that, because implementations should not very in this part.

As soon as we know if we want to use speedzone 8 or speedzone 24, I can provide sampel images for the format.

[willymanilly]

I do my best to live by the KISS principle! I look forward to a community agreed g71 format which I'll add support for. It's great to hear the VICE team are getting involved in that!

My understanding is the intent of g71 is for raw bitstream representation of a 1571 diskette so unsure why the ED and HD formats need to be included in that.

In the meantime, just a thought to incorporate bit removal at the end of tracks, maybe include that number in the byte immediately after the current track data. Considering the offsets for each of the tracks for g64 images are at the start of the image and the actual track length in bytes is at that offset, with the data following, you can simply increase the offset for each track so it can include the extra byte for that purpose. The condition to detect if bits should be removed off the last byte is if the byte directly following the data is nonzero but less than 8, and the offset of the next track is greater than the position of that data, or in the case of the last track there is at least one extra byte at the end of the file. This could be also applied to the standard GCR tracks!

This won't break the g64 format or my zone 8 implementation, which in its current state would simply ignore that extra byte and get the next track data based on the track offset value and the actual track length in the track header.

Example.

Z64Kemptymsdos.g71 (As provided by @Jusalak

00000000 47 43 52 2D 31 35 37 31 00 A8 D4 30 4C 05 00 00 
00000010 00 00 00 00 22 36 00 00 00 00 00 00 F8 66 00 00

00000540 00 00 00 00 00 00 00 00 00 00 00 00 D4 30 92 54 
...
00003620 92 54 D4 30 92 54 92 54 92 54 92 54 92 54 92 54 
...
000066f0 92 54 92 54 92 54 92 54 D4 30 92 54 92 54 92 54

Would become Z64Kemptymsdosmod.zip

00000000 47 43 52 2D 31 35 37 31 00 A8 D4 30 4C 05 00 00 
00000010 00 00 00 00 23 36 00 00 00 00 00 00 FA 66 00 00

00000540 00 00 00 00 00 00 00 00 00 00 00 00 D4 30 92 54
...
00003620 92 54 00 D4 30 92 54 92 54 92 54 92 54 92 54 92
...
000066f0 54 92 54 92 54 92 54 92 54 00 D4 30 92 54 92 54

In summary the actual track size for each track does not change. The updated offsets simply pads a zero before the next track offset location which can be changed to 1-7 if it's required to indicate the last byte of data needs bit removal applied.

Regardless I will support whatever format gets developed and open to adjusting my zone 8 implementation if I get my hands on real world examples of images using those formats.

[markusC64]

My understanding is the intent of g71 is for raw bitstream representation of a 1571 diskette so unsure why the ED and HD formats need to be included in that.

Right. But it's fine to gave all Gxx formats the same (eccept for the 4 byte floppy name in the header), G41, G71, G81, G1M, G2M, G4M (corrsponding to D1M, D2M, D4M). That's IMO the simple part of KISS.

In the meantime, just a thought to incorporate bit removal at the end of tracks, maybe include that number in the byte immediately after the current track data.

That's an idea to look at. With or without my padding - my padding allows some check if the bit removal info is likely present or not.

On the other hand, its simpler to define that speed zones 0 to 15 does neither have support for arbitrary track length nor support for tracks longer than 32767 bytes. And speedzones 16 to 31 must support both. No guessing any more.

Finally it's nice is you have a mixed G71 and current VICE can access the GCR tracks... That's only possible if you do not touch the maximum track size in the header and do not have track length entries that exceed that.

[willymanilly]

I like the simpler approach. IMHO and biased opinion I think my version of the speed zone 8 implementation is the better fit for the g71 format from an emulation perspective for the reasons I previously discussed. I agree images with MFM tracks created from original disks will need arbitrary track length support and setting bit 4 of the speed zone to indicate those ones seems like a good option. As far as I'm aware, only speed zone 0,1,2,3 and 8 will be relevant for 1571 G71 disk images so the unused ones could potentially be redefined for other uses...

Thanks, where possible I do try to respect and support existing file formats and other emulators. I'm always happy to have discussions on file formats where there's bit of a grey area on custom implementations to support things like MFM tracks on G71, where we as a community can agree on standardizing those formats. :)

[markusC64]

As far as I'm aware, only speed zone 0,1,2,3 and 8 will be relevant for 1571 G71 disk images

That's the point I am not sure. There exists a hardware mod for the 1541 (which can quite probable be adjusted for the 1570/1571) that enabled access to speed zones 4 to technically 15, but the higher does not make sense at all. Data arrives that fast that neither a disk can store that reliable not the cpu can send/receive data that fast. The mod is intended to be ale to read double density MFM disk - and that works an a 1541. A side effect of that mod is that speed zones 4 to 7 are available. And it turns out that the speed zones 4 to 6 roughly approximate the speed zones 0 to 3 of an 8250, i.e. it is possible to decode the data of an 8250 with it without errors - similar to the way it works with the neighboring speed zone of a 1541. Regarding that background, they might be needed sometime.

I see. That will be hard work to find a possible flag to flag your version and my version. For my conversion tool, I need to store arbitrary data. The real floppy where it belongs to might only be known after vieweing the data... or never, when it's a system outside the scope - but MFM decoding still works.

One thing we should IMO check: Does VICE make a buffer overflow if we give it a Speed 8 track... Remember that C does not notice and just overwrite neighbour data in memory. That's another reason I did not use track length > 7928 in my approach for storing FM/MFM data - and have chosen a divisor of 8 instead of 4 or even 2 (which would be sufficient not to overflow). Do you agree that we should avoid a fiormat that creates buffer overflows in VICE? And that VICE should be able to read the GCR tracks from those images?

Note that VICE does not support MFM tracks, so it makes no difference if VICE returns this garbage or that garbage. But buffer overflows are really bad since you cannot tell what will happen after that.

[GideonZ]

Making choices in a file format based on a buffer overflow flaw in some program sounds to me as the world backwards. Buffer overflows are the result of poor programming habits, and if we decide to adjust file formats to this, we get poor file formats.

[markusC64]

Sure. But there is no other emulator with such a feature set - e. g. its monitor. So VICE is really needed. And look, they did not implement the MFM extension of the Ultimate also... their standpoint is that such g64 / g71 are invalid and its okay that they create a buffer overflow and likewise. Its the same with g64 images that do not reserve 7928 bytes per track. So every track padded to that value. They think they are invalid and therefore its okay to overwrite some data of the next track in the image file when writing to such images.

It does not help - breaking VICE makes a format unusable to me, and probably to others.

Edit: For a proper format, hope for G64 v2. Perhaps we should meet the VICE developers in their channel to discuss G64 V2. And if we need something for the time in between, do what seems necessary... even if it is not what you wish to make.

[Jusalak]

Interesting discussion.

However, I am having a rather pragmatic viewpoint - what are these MFM images actually good for? In the old days, you could read and write physical floppies of PC and CP/M machines, but as far as I know, .g71 is recognized only by Commodore emulators.

For example, if you could convert msdos formatted .g71 to and from PC 360 kB floppy images, that would give a plenty of stuff to test and inspect.

[markusC64]

CP/M on C128 is documented to support MFM disks. So to run everything on CP/M, you need MFM support. IMHO PC Floppies are just an easy way to test MFM support.

[Jusalak]

Would that kind of PC 360 kB image -> .g71 conversion be feasible? What would it take?

[markusC64]

Except for the part that we are discussing changes to the image format, I have sucessfully read a PC 360k disk with kryoflux and converted it to g71 using g64conv.

For converting an .img file to g71, you would have to build a template file (which is .txt) and use g64conv to do the job.

Edit: Or use the new program which is currently developed and much faster...

Edit 2: Another idea for conversion .img to .g64 / .g71: Use a Gotek together with kryoflux and g64conv.

[markusC64]

@willymanilly One idea - dunno whether it's good or bad.

If we take your suggestion and set the "GCR" text part if the header to "GCR" if no MFM track (or a non padded GCR track) is present and to "GCM" (artifical word of "GCR" and "MFM") otherwise.

Thus VICE won't load the file that makes VICE misbehave. But one can use the successor of 64conv to convert to my format and back, so VICE can be used when necessary.

Then we would not have to care about VICE compatibility and can easyly and reliably device what format it is.

[GideonZ]

Good idea because "GCR-1571" does not describe what it really is; it suggests GCR, but it isn't. MFM-1571 sounds good, but then it shouldn't have GCR tracks, right? :`) So how do we handle mixed disks?

If the disk is purely MFM, do we need to store it in raw format at all?

I do think, however, that having a header and meta data is necessary. The .dxx file formats are rather bad.

[markusC64]

GCR-1571 resp. GCR-1541 is standard. GCM-1571 resp. GCM-1570 for mixed mode is what I proposed. MFM-1571 or MFM-1570 (the single sided version) isn't wrong, too. The question is: Is it worth the extra decision for detecting only MFM tracks? The controller itself allows FM tracks, but the wirering does not. Space for a hardware mod one might want to emulate later... "GCM" seems to fit for that case, too.

And yes, raw format is not bad... Consider a copy protection that explicitly stores checked bytes in the Gaps. An emulator typically has more CPU power available compared to the ultimate. The Read Track command allows to verify them.

On the other hand a faster to process format for hardware that has less CPU power available such as the ultimate is okay. The ultimate already has implemented one.

The really slow g64conv allows to convert between them... with 'old' storing semantics.

[willymanilly]

I'm happy to use GCM-1571 for images with MFM tracks if that helps other emulators.

[markusC64]

I suppose we have a decision. :-)

I assume you mean "GCM-1571" for double sided images. For single sided images, "GCM-1570" might be better. Or even "GCM-1541" if the image has a chance to run on a 1541 (remember I mentioned above there is a MFM reading mod).

[GideonZ]

Remember that a disk format describes the disk. It does not describe the drive. So I would suggest to record meta data explicitly, like number of sides, number of tracks per side, etc.

[markusC64]

Good point to note when asked in discussion for G64/71 V2.

For the time being, its better not to change to much. When formating MFM Tracks (back) to GCR, you have to change too much of the image to be compatible to other emulators. And vice versa.

[markusC64]

I have created a sample image of my understanding from the above conversation.

Note it will only decode without errors if the length correction of the track by reoving 0 to 7 bits is done correctly. bsa-360k.asGCM.g71.zip