search

jfdelnero / HxCFloppyEmulator

HxC Floppy Drive Emulator toolkit
https://hxc2001.com/floppy_drive_emulator/
84 stars 13 forks source link

Multiple failures on .JV3 and .DMK formats (TRS80) #15

Open ByteProject opened 5 months ago

ByteProject commented 5 months ago

I am using the command line tool on Ubuntu Linux. The tool says it supports .JV3 and .DMK but it does not unfortunately. The loaders for the formats are found though. Here the issues I have encountered:

If you could look into this and provide a fix or workaround, that would be much appreciated. I've attached disk images for you as a reference and for testing. These contain my own game "The Ghosts of Blackwood Manor" for TRS80 Model 3 and TRS80 Model 4.

If you have further questions, please let me know. Happy to help!

ghosts_trs80_jv3_dmk.zip

jfdelnero commented 5 months ago

list, getfile, pufile currently only support DOS and AmigaDos partition types. The TRS80 file system isn't support for the moment. I would be happy to add if someone provide me some kind of library to handle this file system.

ByteProject commented 5 months ago

Oh, I see. And I understand. For DMK, there is DMKLIB available. And there also is source available for converting DMK to JV3 and vice versa: dmk2jv3 and jv2dmk. Hope this helps. A good friend of mine, who is a real expert for everything TRS80 related, may want to add something here. Will ping him. But DMKLIB might be a good foundation.

ByteProject commented 5 months ago

And here is a great reference document generally on TRS80 emulator disk formats, this one is more about JV3 but touches DMK as well https://www.tim-mann.org/trs80/dskspec.html and this one goes more into detail regarding DMK http://cpmarchives.classiccmp.org//trs80/mirrors/www.discover-net.net/~dmkeil/trs80/trstech.htm#Technical-DMK-disks

jfdelnero commented 5 months ago

DMK and JV3 are indeed already supported. the dmk/jv3 images conversions are working. This is not an image support issue.

What is missing is the TRS80 file system support to be able to browse into the TRS80 file system.

MBilderbeek commented 3 months ago

In the openMSX emulator we use the DMK format to store lower level information than just the sector data, e.g. to be able to dump disks including their copy protection. This works well for practically all copy protections.

To dump MSX disks, I bought a GreaseWeazle, which can dump to HFE or RAW format. These files I can load into HxCFloppyEmulator and export to DMK in order to use them in openMSX.

Unfortunately, this doesn't seem to work. The created RAW file is fine, as it can be written back to disk and then the disk works. Also, dumping a plain disk (without any special things like copy protections) works fine with the GreaseWeazle. But when I export the dumped RAW or HFE file to DMK with HxCFloppyEmulator, the resulting DMK file seems bogus. It's not working in openMSX and a tool like 'analyze-dmk' sees a lot of issues in the file. (See https://github.com/openMSX/openMSX/tree/master/Contrib/dmk for its source code.)

I'm attaching the RAW files of the dump and the resulting DMK conversion with HxCFloppyEmulator Software version 2.15.3.1 (I used the latest git code).

FAC_demo_4.zip FAC_demo_4.dmk.gz

Although it's a plain jane disk, this is what the DMK contains according to analyze-dmk:

-- physical track 0, head 0
 0: AOfst= 130 C=  0 H=  0 R=  1 N=  2 ACrc=ca6f,ok  data mark not found within 43 bytes from address mark
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 0, head 1
 0: AOfst= 129 C=  0 H=  1 R=  1 N=  2 ACrc=fd5f,ok  data mark not found within 43 bytes from address mark
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 1, head 0
 0: AOfst= 130 C=  1 H=  0 R=  1 N=  2 ACrc=bcdb,ok  data mark not found within 43 bytes from address mark
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 1, head 1

etc. etc.

Is there something wrong with HxCFloppyEmulator or something with analyze-dmk and openMSX regarding the DMK format?

simonowen commented 3 months ago

@MBilderbeek : It looks like that exported DMK image is bad.

Looking at the raw file in a hex editor, it's missing the synchronisation needed to re-frame the bytes correctly for the DMK track data. The start of track gap bytes, IAM and first IDAM are good, but there's no synchronisation after that. A couple of sectors later you can see it's 4 bits out, so the nibbles are swapped:

image

You can just make out the A1A1A1FE of a later IDAM, but the address mark index in the image can't point at the correct address mark byte values needed for a reading application to parse the track.

MBilderbeek commented 3 months ago

@simonowen thanks a lot for analysing this! What is going wrong here then? Have I been doing something wrong? Or are the (3) disks which I tried somehow not suitable to convert from raw to DMK? Or is my dumping incorrect? (But why does putting the dump back on a real disk still give me a working disk then?)

Sorry, I'm not an expert in this (@m9710797 is) - I'd just like to use my GreaseWeazle to dump my MSX disks and convert them to DMK to use in openMSX.

simonowen commented 3 months ago

@MBilderbeek: You're doing nothing wrong and your KryoFlux STREAM files are good (well, except for a possible data CRC error on cyl 79 head 1 sector 7). It's just the conversion to DMK that is missing some synchronisation during the conversion, which meant your DMK disk image was not valid. The HxCFloppyEmulator utility will need to be updated to fix that.

DMK is a track based format that expects all bytes to be perfectly aligned on tracks in the image. There's an index for each track that contains offsets of address marks, and reading bytes at those offsets should show familiar address marks like FE and FB. The analyze-dmk error messages you showed it wasn't recognising the address marks at those offsets, and that's because the bits for each data byte weren't correctly aligned for most sectors. It looked like only the first sector header was aligned in the image.

The STREAM dumps are raw and have no guarantee of bit alignment so it's up to the converter to re-frame the data using the FDC sync markers on the track. That's what a typical FDC does when reading from the track, and what's needed to give valid DMK track data for the image.

MBilderbeek commented 3 months ago

@simonowen thanks for clearing that up. So, I hope @jfdelnero can help me here.... or who ever is able to understand the HxCFloppyEmulator code well enough to improve the code for DMK export. Also @simonowen - are you planning to add export to DMK in samdisk? :) (We've talked about this a long time ago, but now it's quite relevant for me, now that I have a GreaseWeazle to make flux dumps!)

jfdelnero commented 3 months ago

Hi,

This is a known issue and it's even commented into the sources since several years :

https://github.com/jfdelnero/HxCFloppyEmulator/blob/main/libhxcfe/sources/loaders/dmk_loader/dmk_writer.c

// Byte align the track...
// TODO : Align each sectors ID Mark and Data Mark individually
//        (Needed for stream based images sources)

Since there are some track modification functions/helpers available now to insert / remove cells into tracks, this should be easier to do. Will try to take some minutes to have a look again to this issue.

MBilderbeek commented 3 months ago

Aha, okay. Would you pretty please try to fix the TODO? Or is there any other tool out there to convert my flux stream dumps to DMK?

jfdelnero commented 3 months ago

yes i said in my previous message i will.

(but i think that something else should be used and this dmk format should be avoided today - this alignment constraint is a no-go from a preservation point of view : you have to modify the tracks content on multiple points to make it work... )

MBilderbeek commented 3 months ago

Ah, sorry, I replied to the mail I got of the comment and that one didn't have these last two lines! I'd be happy to test it with my MSX disks. Feel free to trigger me anytime when there's something I can test.

MBilderbeek commented 1 month ago

Any news on this so far? Anything I can do to help?

MBilderbeek commented 1 week ago

(but i think that something else should be used and this dmk format should be avoided today - this alignment constraint is a no-go from a preservation point of view : you have to modify the tracks content on multiple points to make it work... )

Would HFE be a better format, being only about twice the size? Anyways, still hoping you're willing to take a look at this, as we haven't got HFE in our emulator for a while yet to come.

jfdelnero commented 1 week ago

@MBilderbeek Please try this version and report if this is working properly, i have done the alignment modifications : [REMOVED]

MBilderbeek commented 1 week ago

@jfdelnero can you please send me the source code or a patch to the current master? I am on Linux so I can't use your binaries.

Oh, thanks a lot of course!!

jfdelnero commented 1 week ago

@jfdelnero can you please send me the source code or a patch to the current master? I am on Linux so I can't use your binaries.

Oh, thanks a lot of course!!

This is already on the current master.

MBilderbeek commented 1 week ago

@jfdelnero can you please send me the source code or a patch to the current master? I am on Linux so I can't use your binaries. Oh, thanks a lot of course!!

This is already on the current master.

Perfect! Will try it out.

MBilderbeek commented 1 week ago

OK, @jfdelnero - trying HxC Floppy Emulator : Floppy image file converter v2.15.8.2.

I used the data from https://github.com/jfdelnero/HxCFloppyEmulator/issues/15#issuecomment-1995278161 as input.

I exported the raw dump as DMK file again and ran analyze-dmk again, but the aforementioned analyze-dmk says:

-- physical track 0, head 0
 0: AOfst= 130 C=  0 H=  0 R=  1 N=  2 ACrc=ca6f,ok  DOfst= 175 T=n DCrc=9473,ok 
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 0, head 1
 0: AOfst= 129 C=  0 H=  1 R=  1 N=  2 ACrc=fd5f,ok  DOfst= 174 T=n DCrc=da6e,ok 
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: AOfst=5202 C=  0 H=  1 R=  9 N=  2 ACrc=74f6,ok  data mark not found within 43 bytes from address mark
-- physical track 1, head 0
 0: AOfst= 130 C=  1 H=  0 R=  1 N=  2 ACrc=bcdb,ok  DOfst= 175 T=n DCrc=3634,ok 
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: AOfst=2667 C=  1 H=  0 R=  5 N=  2 ACrc=701f,ok  data mark not found within 43 bytes from address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 1, head 1
 0: AOfst= 130 C=  1 H=  1 R=  1 N=  2 ACrc=8beb,ok  DOfst= 175 T=n DCrc=db81,ok 
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 2, head 0
 0: AOfst= 130 C=  2 H=  0 R=  1 N=  2 ACrc=2707,ok  DOfst= 175 T=n DCrc=66dd,ok 
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: AOfst=1397 C=  2 H=  0 R=  3 N=  2 ACrc=4165,ok  data mark not found within 43 bytes from address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark
-- physical track 2, head 1
 0: AOfst= 129 C=  2 H=  1 R=  1 N=  2 ACrc=1037,ok  DOfst= 174 T=n DCrc=bb8b,ok 
 1: ... skipping wrong IDAM entry, does not point to an address mark
 2: ... skipping wrong IDAM entry, does not point to an address mark
 3: ... skipping wrong IDAM entry, does not point to an address mark
 4: ... skipping wrong IDAM entry, does not point to an address mark
 5: ... skipping wrong IDAM entry, does not point to an address mark
 6: ... skipping wrong IDAM entry, does not point to an address mark
 7: ... skipping wrong IDAM entry, does not point to an address mark
 8: ... skipping wrong IDAM entry, does not point to an address mark

etc.etc.

And indeed, the image doesn't work on openMSX.

In other words: this is still not a valid DMK image. What did I do wrong?

jfdelnero commented 1 week ago

Can you try to create convert the attached hfe to dmk and analyze the file with analyze-dmk ?

Empty Floppy.hfe.zip

MBilderbeek commented 1 week ago

Can you try to create convert the attached hfe to dmk and analyze the file with analyze-dmk ?

Empty Floppy.hfe.zip

As openMSX doesn't support single density disk images, analyze-dmk also doesn't like it:

-- physical track 0, head 0 0: ... skipping single-density sector 1: ... skipping single-density sector 2: ... skipping single-density sector 3: ... skipping single-density sector 4: ... skipping single-density sector 5: ... skipping single-density sector 6: ... skipping single-density sector 7: ... skipping single-density sector 8: ... skipping single-density sector

However, I tried with the empty "DSK" image (created by openMSX itself) and analyze-dmk then gives this expected output after conversion to DMK with HxCFloppyEmulator:

-- physical track 0, head 0
 0: AOfst= 158 C=  0 H=  0 R=  1 N=  2 ACrc=ca6f,ok  DOfst= 202 T=n DCrc=3e2f,ok 
 1: AOfst= 792 C=  0 H=  0 R=  2 N=  2 ACrc=9f3c,ok  DOfst= 836 T=n DCrc=6353,ok 
 2: AOfst=1426 C=  0 H=  0 R=  3 N=  2 ACrc=ac0d,ok  DOfst=1470 T=n DCrc=da6e,ok 
 3: AOfst=2060 C=  0 H=  0 R=  4 N=  2 ACrc=359a,ok  DOfst=2104 T=n DCrc=da6e,ok 
 4: AOfst=2694 C=  0 H=  0 R=  5 N=  2 ACrc=06ab,ok  DOfst=2738 T=n DCrc=6353,ok 
 5: AOfst=3328 C=  0 H=  0 R=  6 N=  2 ACrc=53f8,ok  DOfst=3372 T=n DCrc=da6e,ok 
 6: AOfst=3962 C=  0 H=  0 R=  7 N=  2 ACrc=60c9,ok  DOfst=4006 T=n DCrc=da6e,ok 
 7: AOfst=4596 C=  0 H=  0 R=  8 N=  2 ACrc=70f7,ok  DOfst=4640 T=n DCrc=da6e,ok 
 8: AOfst=5230 C=  0 H=  0 R=  9 N=  2 ACrc=43c6,ok  DOfst=5274 T=n DCrc=da6e,ok 
-- physical track 0, head 1
 0: AOfst= 158 C=  0 H=  1 R=  1 N=  2 ACrc=fd5f,ok  DOfst= 202 T=n DCrc=da6e,ok 
 1: AOfst= 792 C=  0 H=  1 R=  2 N=  2 ACrc=a80c,ok  DOfst= 836 T=n DCrc=da6e,ok 
 2: AOfst=1426 C=  0 H=  1 R=  3 N=  2 ACrc=9b3d,ok  DOfst=1470 T=n DCrc=da6e,ok 
 3: AOfst=2060 C=  0 H=  1 R=  4 N=  2 ACrc=02aa,ok  DOfst=2104 T=n DCrc=da6e,ok 
 4: AOfst=2694 C=  0 H=  1 R=  5 N=  2 ACrc=319b,ok  DOfst=2738 T=n DCrc=da6e,ok 
 5: AOfst=3328 C=  0 H=  1 R=  6 N=  2 ACrc=64c8,ok  DOfst=3372 T=n DCrc=c40b,ok 
 6: AOfst=3962 C=  0 H=  1 R=  7 N=  2 ACrc=57f9,ok  DOfst=4006 T=n DCrc=c40b,ok 
 7: AOfst=4596 C=  0 H=  1 R=  8 N=  2 ACrc=47c7,ok  DOfst=4640 T=n DCrc=c40b,ok 
 8: AOfst=5230 C=  0 H=  1 R=  9 N=  2 ACrc=74f6,ok  DOfst=5274 T=n DCrc=c40b,ok 
-- physical track 1, head 0
 0: AOfst= 158 C=  1 H=  0 R=  1 N=  2 ACrc=bcdb,ok  DOfst= 202 T=n DCrc=c40b,ok 
 1: AOfst= 792 C=  1 H=  0 R=  2 N=  2 ACrc=e988,ok  DOfst= 836 T=n DCrc=c40b,ok 
 2: AOfst=1426 C=  1 H=  0 R=  3 N=  2 ACrc=dab9,ok  DOfst=1470 T=n DCrc=c40b,ok 
 3: AOfst=2060 C=  1 H=  0 R=  4 N=  2 ACrc=432e,ok  DOfst=2104 T=n DCrc=c40b,ok 
 4: AOfst=2694 C=  1 H=  0 R=  5 N=  2 ACrc=701f,ok  DOfst=2738 T=n DCrc=c40b,ok 
 5: AOfst=3328 C=  1 H=  0 R=  6 N=  2 ACrc=254c,ok  DOfst=3372 T=n DCrc=c40b,ok 
 6: AOfst=3962 C=  1 H=  0 R=  7 N=  2 ACrc=167d,ok  DOfst=4006 T=n DCrc=c40b,ok 
 7: AOfst=4596 C=  1 H=  0 R=  8 N=  2 ACrc=0643,ok  DOfst=4640 T=n DCrc=c40b,ok 
 8: AOfst=5230 C=  1 H=  0 R=  9 N=  2 ACrc=3572,ok  DOfst=5274 T=n DCrc=c40b,ok 
-- physical track 1, head 1
 0: AOfst= 158 C=  1 H=  1 R=  1 N=  2 ACrc=8beb,ok  DOfst= 202 T=n DCrc=c40b,ok 
 1: AOfst= 792 C=  1 H=  1 R=  2 N=  2 ACrc=deb8,ok  DOfst= 836 T=n DCrc=c40b,ok 
 2: AOfst=1426 C=  1 H=  1 R=  3 N=  2 ACrc=ed89,ok  DOfst=1470 T=n DCrc=c40b,ok 
 3: AOfst=2060 C=  1 H=  1 R=  4 N=  2 ACrc=741e,ok  DOfst=2104 T=n DCrc=c40b,ok 
 4: AOfst=2694 C=  1 H=  1 R=  5 N=  2 ACrc=472f,ok  DOfst=2738 T=n DCrc=c40b,ok 
 5: AOfst=3328 C=  1 H=  1 R=  6 N=  2 ACrc=127c,ok  DOfst=3372 T=n DCrc=c40b,ok 
 6: AOfst=3962 C=  1 H=  1 R=  7 N=  2 ACrc=214d,ok  DOfst=4006 T=n DCrc=c40b,ok 
 7: AOfst=4596 C=  1 H=  1 R=  8 N=  2 ACrc=3173,ok  DOfst=4640 T=n DCrc=c40b,ok 
 8: AOfst=5230 C=  1 H=  1 R=  9 N=  2 ACrc=0242,ok  DOfst=5274 T=n DCrc=c40b,ok

empty_dsk.zip

jfdelnero commented 1 week ago

Can you share a non working stream dump ?

MBilderbeek commented 6 days ago

Can you share a non working stream dump ?

Is the raw dump not good enough, which I provided in my original post? If not, please specify how I make a suitable dump with my GreaseWeazle.

jfdelnero commented 4 days ago

Your raw is indeed fine, and this was a silly mistake in my last modifications. This now appears to work fine with your raw dump. Please try it and let me know how it works for you.

MBilderbeek commented 4 days ago

Yes, this seems to work fine with 2 cases I tried so far! Thanks a lot! This enables a lot for me!!

I'll try some more complex ones, with copy protections and all. I'll report here if I run into problems.

jfdelnero commented 4 days ago

( beta version released with the modifications : https://hxc2001.com/download/floppy_drive_emulator/HxCFloppyEmulator_soft_beta.zip )