[Issue] Redump Dreamcast ROMs fail to verify after conversion to CHD format.

[TheRealGusBus]

Summary

Adding a fresh ZIP file with bin/cue files to the ROM set verifies fine. However, conversion to CHD and running the check-roms command after results in the CHD file silently being moved to the Trash directory.

Steps Taken to Reproduce

Starting off with a fresh ROM directory:

gus@gus-desktop-nuc:/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce $ oxyromon purge-roms -om
Processing missing ROM files
Deleted 1 missing ROM file(s) from the database
Processing orphan ROM files

Importing single Dreamcast game:

gus@gus-desktop-nuc:/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce $ oxyromon import-roms WWF\ Royal\ Rumble\ \(USA\).zip 
Processing "WWF Royal Rumble (USA).zip"
Processing "WWF Royal Rumble (USA) (Track 1).bin (WWF Royal Rumble (USA).zip)"
Matches "WWF Royal Rumble (USA) (Track 1).bin"
Processing "WWF Royal Rumble (USA) (Track 2).bin (WWF Royal Rumble (USA).zip)"
Matches "WWF Royal Rumble (USA) (Track 2).bin"
Processing "WWF Royal Rumble (USA) (Track 3).bin (WWF Royal Rumble (USA).zip)"
Matches "WWF Royal Rumble (USA) (Track 3).bin"
Processing "WWF Royal Rumble (USA).cue (WWF Royal Rumble (USA).zip)"
Matches "WWF Royal Rumble (USA).cue"
Moving to "/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce/verified/Sega - Dreamcast/WWF Royal Rumble (USA).zip"

Convert Dreamcast set to CHD:

gus@gus-desktop-nuc:/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce $ oxyromon convert-roms 
Please select systems: Sega - Dreamcast
Please select a format: CHD
Processing "Sega - Dreamcast"
Extracting "WWF Royal Rumble (USA) (Track 1).bin"
Extracting "WWF Royal Rumble (USA) (Track 2).bin"
Extracting "WWF Royal Rumble (USA) (Track 3).bin"
Extracting "WWF Royal Rumble (USA).cue"
Creating "WWF Royal Rumble (USA).chd"
Copying to "/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce/verified/Sega - Dreamcast/WWF Royal Rumble (USA).cue"
Deleting "/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce/verified/Sega - Dreamcast/WWF Royal Rumble (USA).zip"

Verify contents of directory:

gus@gus-desktop-nuc:/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce $ ls -al verified/Sega\ -\ Dreamcast 
total 144132
drwxrwxr-x  2 gus gus    229376 Jan  2 20:15  .
drwxrwxr-x 43 gus gus      4096 Dec 26 22:15  ..
-rw-rw-r--  1 gus gus 147347006 Jan  2 20:15 'WWF Royal Rumble (USA).chd'
-rw-rw-r--  1 gus gus       360 Jan  2 20:15 'WWF Royal Rumble (USA).cue'

Run check-roms command against Dreamcast set:

gus@gus-desktop-nuc:/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce $ oxyromon check-roms
Please select systems: Sega - Dreamcast
Processing "Sega - Dreamcast"
Processing "WWF Royal Rumble (USA).chd"
Extracting "WWF Royal Rumble (USA).chd"
Processing "WWF Royal Rumble (USA).cue"

At this point:

1) the web server shows the game as complete:

2) The CHD file has been moved to the Trash:

gus@gus-desktop-nuc:/media/gus/172106f2-d2bf-4ebb-af09-dfa9a79d7cce $ ls -R verified/Sega\ -\ Dreamcast
'verified/Sega - Dreamcast':
 Trash  'WWF Royal Rumble (USA).cue'

'verified/Sega - Dreamcast/Trash':
'WWF Royal Rumble (USA).chd'

[alucryd]

Thanks for the detailed steps, trying to reproduce as I write.

Edit: Good news (well, sort of), I can reproduce with that particular game! Looking into it.

[alucryd]

So it looks like an issue with chdman, some dreamcast games still can't be fully reverted apparently, in the case of WWF the resulting bin is missing 352800 bytes. I don't have many Dreamcast games, and the ones I do have are all fine so I didn't catch that. Unfortunately there's no obvious difference between one that works and one that doesn't. I'll open a bug report upstream.

There was also a typo on my end making rom deletion silent in check-roms, this is now fixed on the develop branch, deletion will be clearly visible now.

[alucryd]

Issue opened upstream: https://github.com/mamedev/mame/issues/11903

My testing seems to indicate that chdman is removing some padding at the beginning of track 2, the size difference is exactly the number of missing 0.

If that's really the issue, and is the same for all files, I could probably repair them in oxyromon, but let's wait to hear from the mame devs before I start anything.

[TheRealGusBus]

Are the ones that test okay for you unlicensed games? I tested against a nearly full Dreamcast set and everything but the unlicensed/MIL-CD games were moved to the trash.

[alucryd]

Both sonic games, shenmue and ecco USA releases verify correctly.

According to upstream, this is actually intended, chdman strips a pregap because they use the TOSEC layout which is different from the one Redump uses.

Not sure what the best course of action is, I'll try to reach out to redump.

[alucryd]

New piece of information from Redump, the gap is there for accuracy and can contain actual data. I'm temporarily disabling CHD for Dreamcast, even if I could "repair" most games by padding them with a few zeroes, I can't recreate any lost data so the only foolproof solution would be for chdman to stop stripping these gaps. Pinning the issue.

[alucryd]

@TheRealGusBus If you're on x86_64 linux, would you be able to give this modified chdman a try?

https://files.alucryd.xyz/chdman

It works fine on WWF and other games that failed otherwise on my end. Will submit a PR upstream tomorrow if that works for you too.

[TylerVigario]

I can confirm that my Dreamcast ROMs are being trashed. I'd like to know what CHDMAN platforms follow TOSEC standards and are affected.

oxyromon check-roms
Please select systems: Sega - Dreamcast
Processing "Sega - Dreamcast"
Processing "18 Wheeler - American Pro Trucker (USA).cue"
Processing "18 Wheeler - American Pro Trucker (USA).chd"
Extracting "18 Wheeler - American Pro Trucker (USA).chd"
Creating "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash"
Moving to "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash/18 Wheeler - American Pro Trucker (USA).chd"
Processing "AeroWings (USA).chd"
Extracting "AeroWings (USA).chd"
Moving to "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash/AeroWings (USA).chd"
Processing "AeroWings 2 - Airstrike (USA).chd"
Extracting "AeroWings 2 - Airstrike (USA).chd"
Moving to "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash/AeroWings 2 - Airstrike (USA).chd"

[TheRealGusBus]

@alucryd I did a release upgrade to the latest Ubuntu LTS, but it seems the libraries are still too old:

./chdman: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ./chdman)
./chdman: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./chdman)

ldd --version shows glibc 2.35.

I can try building chdman on this system to verify with some basic instructions.

[alucryd]

Ah, sorry about that, I run Arch everywhere. I'll push the code later today so you can try building it. It's not difficult, but it might take some time as mame has a huge codebase.

[alucryd]

I can confirm that my Dreamcast ROMs are being trashed. I'd like to know what CHDMAN platforms follow TOSEC standards and are affected.

oxyromon check-roms
Please select systems: Sega - Dreamcast
Processing "Sega - Dreamcast"
Processing "18 Wheeler - American Pro Trucker (USA).cue"
Processing "18 Wheeler - American Pro Trucker (USA).chd"
Extracting "18 Wheeler - American Pro Trucker (USA).chd"
Creating "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash"
Moving to "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash/18 Wheeler - American Pro Trucker (USA).chd"
Processing "AeroWings (USA).chd"
Extracting "AeroWings (USA).chd"
Moving to "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash/AeroWings (USA).chd"
Processing "AeroWings 2 - Airstrike (USA).chd"
Extracting "AeroWings 2 - Airstrike (USA).chd"
Moving to "/mnt/storage/Nextcloud/__groupfolders/1/Roms/Sega - Dreamcast/Trash/AeroWings 2 - Airstrike (USA).chd"

As far as I can tell, it's only in the GD-Rom code, so hopefully this only affects Dreamcast.

[alucryd]

It is fixable for the most part, but if you guys still have the BINs or can grab them from the website that records it all I'd rather spend time on the next item in my list.

I will be tackling an export-roms command, similar to convert-roms, but the goal is to export all or a subset of a system to a specified target, be it as simple as a directory, or something more complex like network upload to OPL on the PS2 (which I'll probably do first since I just got a PS2 to do just that). One huge difference is that export-roms will support lossy formats as well since it will leave the originals untouched.

[alucryd]

This is my branch with the fix: https://github.com/alucryd/mame/tree/dreamcast-keep-pregap

Run the following at the root: make -j4 VERBOSE=1 NOWERROR=1 OPTIMIZE=2 TOOLS=1 ARCHOPTS=-flifetime-dse=1.

You can adjust -j4 with the number of cpu cores you have. On Ubuntu you'll need to install build-essentials and a few development packages, I believe they all end with -dev there. You can refer to the makedepends array of the Arch recipe [0] for the required dependencies, or just compile and wait until you get a missing header error, install it, rinse and repeat. Once everything is built, you should have a chdman binary at the root. Note that it will take a while to build.

[0] https://gitlab.archlinux.org/archlinux/packaging/packages/mame/-/blob/main/PKGBUILD?ref_type=heads

[alucryd]

The fix has been merged upstream. Not sure when their next release is due, but I'll require 0.262 preemptively in the code.

[TheRealGusBus]

It is fixable for the most part, but if you guys still have the BINs or can grab them from the website that records it all I'd rather spend time on the next item in my list.

I will be tackling an export-roms command, similar to convert-roms, but the goal is to export all or a subset of a system to a specified target, be it as simple as a directory, or something more complex like network upload to OPL on the PS2 (which I'll probably do first since I just got a PS2 to do just that). One huge difference is that export-roms will support lossy formats as well since it will leave the originals untouched.

At least for me, this is low priority since the CHD files still work in emulators, they just don't verify in oxyromon.

Also, just to clarify, does this fix decompression of the CHD files so that existing CHDs will verify properly, or just newly created CHD files?

[alucryd]

The upstream fix isn't retroactive, only new CHDs will be fine. Fixing the broken files would require additional work on my end as the gaps are already lost.

[TheRealGusBus]

So I've installed libsdl2-dev and libsdl1.2-dev. qtbase5-dev was already installed. I still can't compile for some reason:

 $ make -j1 VERBOSE=1NOWERROR=1OPTIMIZE=2 TOOLS=1 ARCHOPTS=-flifetime-dse=1
GCC 11.4.0 detected
fatal: No names found, cannot describe anything.
make -R verbose=1 -C build/projects/sdl/mame/gmake-linux config=release64 precompile
make[1]: Entering directory '/home/gus/temp/mame/build/projects/sdl/mame/gmake-linux'
make[1]: Leaving directory '/home/gus/temp/mame/build/projects/sdl/mame/gmake-linux'
make -R verbose=1 -C build/projects/sdl/mame/gmake-linux config=release64
make[1]: Entering directory '/home/gus/temp/mame/build/projects/sdl/mame/gmake-linux'
Linking chdman...
g++ -o ../../../../../chdman ../../../../linux_gcc/obj/x64/Release/src/tools/chdman.o ../../../../linux_gcc/obj/x64/Release/generated/version.o     -L"../../../../../../../../../usr/X11/lib" -L"../../../../../../../../../usr/X11R6/lib" -L"../../../../../../../../../usr/openwin/lib" -L"." -L"../../../../linux_gcc/bin/x64/Release" -L"../../../../linux_gcc/bin/x64/Release/mame_mame" -m64 -flifetime-dse=1 -Wl,--start-group ../../../../linux_gcc/bin/x64/Release/libutils.a ../../../../linux_gcc/bin/x64/Release/libexpat.a ../../../../linux_gcc/bin/x64/Release/lib7z.a ../../../../linux_gcc/bin/x64/Release/mame_mame/libocore_sdl.a ../../../../linux_gcc/bin/x64/Release/libzlib.a ../../../../linux_gcc/bin/x64/Release/libzstd.a ../../../../linux_gcc/bin/x64/Release/libflac.a ../../../../linux_gcc/bin/x64/Release/libutf8proc.a -ldl -lrt -lm -lpthread -lutil -Wl,--end-group
/usr/bin/ld: ../../../../linux_gcc/bin/x64/Release/mame_mame/libocore_sdl.a(osdlib_unix.o): in function `osd_get_clipboard_text[abi:cxx11]()':
osdlib_unix.cpp:(.text+0x262): undefined reference to `SDL_HasClipboardText'
/usr/bin/ld: osdlib_unix.cpp:(.text+0x279): undefined reference to `SDL_GetClipboardText'
/usr/bin/ld: osdlib_unix.cpp:(.text+0x2a6): undefined reference to `SDL_free'
/usr/bin/ld: ../../../../linux_gcc/bin/x64/Release/mame_mame/libocore_sdl.a(osdlib_unix.o): in function `osd_set_clipboard_text(std::basic_string_view<char, std::char_traits<char> >)':
osdlib_unix.cpp:(.text+0x340): undefined reference to `SDL_SetClipboardText'
collect2: error: ld returned 1 exit status
make[2]: *** [chdman.make:261: ../../../../../chdman] Error 1
make[1]: *** [Makefile:1114: chdman] Error 2
make[1]: Leaving directory '/home/gus/temp/mame/build/projects/sdl/mame/gmake-linux'
make: *** [makefile:1290: linux_x64] Error 2

[alucryd]

On my system, these come from the SDL2/SDL_clipboard.h header. Maybe libsdl2 is split into multiple packages on Ubuntu and you're missing the right one?

Edit: According to the package search, it's part of libsdl2-dev :/ https://packages.ubuntu.com/search?searchon=contents&keywords=SDL_clipboard.h&mode=exactfilename&suite=jammy&arch=any

It seems there were missing spaces in the command I gave you, sorry about that, I've updated it, please let me know how it goes. You might need to run make clean before trying again.

[alucryd]

You can also try to run sed -e 's|\# USE_SYSTEM_LIB|USE_SYSTEM_LIB|g' -i makefile to force the use of system libs.

[TheRealGusBus]

Well, it's there already, so I'm more confused than I was.

gus@gus-desktop-nuc:/home/gus $ dpkg-query -L libsdl2-dev | grep clip
/usr/include/SDL2/SDL_clipboard.h
gus@gus-desktop-nuc:/home/gus $ file /usr/include/SDL2/SDL_clipboard.h 
/usr/include/SDL2/SDL_clipboard.h: C source, ASCII text

Edit: It seems to be compiling fine so far using the fixed command.

[TheRealGusBus]

So there's good news and bad news. The good is that the changes to chdman produce a reversible .chd file that's verified by oxyromon. The bad news is that at least the emulator Flycast doesn't launch the game now. It instead gives the error "Unsupported subtype or pre/post gap".

[TheRealGusBus]

I wonder if instead of modifying chdman to keep the gap data, it should simply pad out the data when extracting. I think that prevents the new gap data in a CHD from breaking games in existing applications, and should allow oxyromon to verify the CHD files just fine.

111 should ensure that any games with gap data don't get converted to CHD in the first place, so zero-padding converted CHD files should be a non-issue for bin/cues already managed by oxyromon.

[TylerVigario]

So there's good news and bad news. The good is that the changes to chdman produce a reversible .chd file that's verified by oxyromon. The bad news is that at least the emulator Flycast doesn't launch the game now. It instead gives the error "Unsupported subtype or pre/post gap".

Isn't this just a downstream issue with the fundamental change?

[TheRealGusBus]

@TylerVigario Yeah, scratch my idea of padding it. The flycast repository already has an issue to add support and this is better for preservation purposes.

[maxexcloo]

The upstream fix isn't retroactive, only new CHDs will be fine. Fixing the broken files would require additional work on my end as the gaps are already lost.

I have a bunch of Dreamcast CHD files that don't match currently - are they toast? Probably not a huge deal but just wondering :)

[alucryd]

If you guys have time to build and test, there's a new PR upstream that should fix the fix: https://github.com/mamedev/mame/pull/12087

[alucryd]

The upstream fix isn't retroactive, only new CHDs will be fine. Fixing the broken files would require additional work on my end as the gaps are already lost.

I have a bunch of Dreamcast CHD files that don't match currently - are they toast? Probably not a huge deal but just wondering :)

Afraid so, but maybe the above PR will say otherwise.

[TheRealGusBus]

I've managed to check the type2 and type3 games listed here https://github.com/mamedev/mame/pull/12087#issuecomment-1975098822

All seem fine according to oxyromon.

[TylerVigario]

This seems to be fully resolved on our end

[TheRealGusBus]

I'm only up to the M's in the Dreamcast set, but I haven't seen any errors when converting the bin/cue file to CHD with the -c option enabled.

[alucryd]

I'll update the minimum version requirement to 0.264 in the code then, thanks for all the testing guys.