Import content issue when scanning mame romsets with CHD

[ghost]

First and foremost consider this: Only RetroArch bugs should be filed here. Not core bugs or game bugs This is not a forum or a help section, this is strictly developer oriented Description Retroarch is stuck doing something (cpu usage is above 50%-slow pc here) and log or OSD shows no update while scanning mame romsets with CHD.

Expected behavior scanning should continue, regardless of the romsets were verified or not.

Actual behavior as stated above, when scanning mame romsets with chd, retroarch is stuck and it wont continue scanning. temporarily removing the chd files resolves this but the chd should not be scanned in these case since a separate zip file is associated with the chd instead.

Steps to reproduce the bug just scan a full romset of mame which includes chd files retroarch should stop scanning, and no osd or log info is shown as to whatever is going on. Bisect Results [Try to bisect and tell us when this started happening]

Version/Commit You can find this information under Information/System Information

RetroArch: 1.7.1 0566b05 Environment information OS: Arch Linux 4.14 Compiler: gcc 7.3.0

[fotisandstuff]

I've been having this issue for months. Please, somebody help.

[i30817]

This is probably a duplicate of https://github.com/libretro/RetroArch/issues/6553

[orbea]

Might be the same issue as @i30817 mentioned, but I'm not sure and unfortunately I don't have many chd files to test. The few I do have don't have this problem.

[ghost]

basically the same, but since this thread was create Mar 4, his post is then duplicate of this. but whatever, just saying...

[orbea]

Yea, you're right.

[i30817]

Yeah, it's a duplicate but mine had a bit more info. But i can repeat it here: the issue is that there are some chd in the MAME set that aren't cds but hdd, and retroarch crashes when scanning these. Not very important info but hey.

[ghost]

im not sure how you got backtrace but i dont get a gdb error or crash. scanning just stops for no reason and any rescanning will not work untill retroarch is reset.

[INFO] Comparing with known magic numbers...
[INFO] Could not find compatible system.

it just shows that and nothing more, could probably force it to do something else but its still the same problem, it just wont scan until the CHD's are moved (it shouldnt matter if these were legit or valid chd's)

[orbea]

Yes of course, adding important information here can help especially when the extra noise is avoided.

[i30817]

I don't remember exactly, but i think it wasn't by backtrace but simple testing. We tracked down that the difference was the type of chd from chdman info and i tested i could scan the mame collection without those chds (with other bugs interfering from it being a split set and some kind of non-zero'ed structure causing duplicate entries) and always crashed with any one of them.

In fact, i think scanning chd's isn't even necessary for MAME scans because what identifies a MAME game is the main zip (unless you want to run it in reicast). Thus i could move the subdirectories containing chd out, scan, and then move them back in to get a MAME playlist (with those other bugs mentioned). I don't mind this because i never thought Retroarch scanner job should be to validate the games, just uniquely identify a non-malicious unique sig for metadata.

Basically i think this bug directly comes from code assuming libchd is reading a cd image, either inside libchd itself, or from where retroarch uses it.

[ghost]

CHD scanning was added a few months ago, but that probably ddnt factor in different CHD versions available and that CHD are not always CD-based images or such to begin with.

In any case, regardless if it takes a longer time to scan, it should skip invalid or no matched files(chd in this matter) and proceed to next file to scan. not get stuck in an endless loop

[i30817]

In linux there wasn't a loop, just a crash (segfault). I think it's the same and windows just behaves differently.

[ghost]

i am on linux, if that wasnt clear already on the initial post.

[i30817]

This is the backtrace (from a self compiled Retroarch, reading the tekken4 chd):

Thread 1 "retroarch" received signal SIGSEGV, Segmentation fault.
0x0000555555a1f1e7 in chd_open_file.part ()
(gdb) bt
#0  0x0000555555a1f1e7 in chd_open_file.part ()
#1  0x0000555555a20073 in chd_open ()
#2  0x0000555555a217c2 in chdstream_open ()
#3  0x00005555555c82f3 in intfstream_open_chd_track ()
#4  0x00005555556afc02 in task_database_handler ()
#5  0x00005555555bb745 in retro_task_regular_gather ()
#6  0x00005555555a3225 in rarch_main ()
#7  0x00007fffefe00b97 in __libc_start_main (main=
    0x55555559f600 <main>, argc=1, argv=0x7fffffffdf48, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdf38)
    at ../csu/libc-start.c:310
#8  0x000055555559fd2a in _start ()
(gdb) 

If you want any testing please tell me (although i'm not a gdb wizard, a maybe i should have enabled further debug symbols for retroarch?)

[ghost]

nah just redo the scanning with debug symbols enabled. my 1st chd entries are area51 stuff so that probably ddnt cause any crash but still is the same issue. too lazy now to move stuff frm hdd with very limited space atm.

[i30817]

Ok, a debug bt:

Thread 1 "retroarch" received signal SIGSEGV, Segmentation fault.
0x0000555555bf53fa in chd_close (chd=0x5555577a0020)
    at libretro-common/formats/libchdr/libchdr_chd.c:1030
1030             switch (chd->codecintf[i]->compression)
(gdb) bt
#0  0x0000555555bf53fa in chd_close (chd=0x5555577a0020)
    at libretro-common/formats/libchdr/libchdr_chd.c:1030
#1  0x0000555555bf5199 in chd_open_file (file=0x555557185480, mode=1, parent=0x0, chd=0x7fffffffda98) at libretro-common/formats/libchdr/libchdr_chd.c:929
#2  0x0000555555bf5210 in chd_open (filename=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", mode=1, parent=0x0, chd=0x7fffffffda98)
    at libretro-common/formats/libchdr/libchdr_chd.c:966
#3  0x0000555555bf8a38 in chdstream_open (path=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", track=-1)
    at libretro-common/streams/chd_stream.c:209
#4  0x0000555555603388 in intfstream_open (intf=
    0x5555572f7540, path=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", mode=1, hints=0) at libretro-common/streams/interface_stream.c:127
#5  0x0000555555603b59 in intfstream_open_chd_track (path=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", mode=1, hints=0, track=-1)
    at libretro-common/streams/interface_stream.c:476
#6  0x000055555577ee6c in task_database_chd_get_serial (name=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", serial=0x555557792f87 "")
    at tasks/task_database.c:327
#7  0x000055555577fb16 in task_database_iterate_playlist (db_state=0x555557792d68, db=
    0x555557793fa0, name=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd") at tasks/task_database.c:660
---Type <return> to continue, or q <return> to quit---
#8  0x0000555555780a04 in task_database_iterate (_db=0x555557792d40, db_state=0x555557792d68, db=0x555557793fa0)
    at tasks/task_database.c:1115
#9  0x0000555555780e9d in task_database_handler (task=0x5555572f2ee0) at tasks/task_database.c:1244
#10 0x00005555555f4187 in retro_task_regular_gather () at libretro-common/queues/task_queue.c:182
#11 0x00005555555f4bca in task_queue_check () at libretro-common/queues/task_queue.c:609
#12 0x000055555572099c in ui_application_qt_run(void*) (args=0x0) at ui/drivers/qt/ui_qt_application.cpp:163
#13 0x00005555555d65fb in rarch_main (argc=1, argv=0x7fffffffdf48, data=0x0) at frontend/frontend.c:157
#14 0x00005555557209fb in main(int, char**) (argc=1, argv=0x7fffffffdf48) at ui/drivers/qt/ui_qt_application.cpp:182
(gdb) 

The problem is either in libchd itself or in this part with Retroarch 'assuming' a chd has a track to search a serial (when it's a hdd image in this case):

#5  0x0000555555603b59 in intfstream_open_chd_track (path=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", mode=1, hints=0, track=-1)
    at libretro-common/streams/interface_stream.c:476
#6  0x000055555577ee6c in task_database_chd_get_serial (name=0x5555572f79e0 "/media/i30817/Games/Arcade/tekken4/tef1dvd0.chd", serial=0x555557792f87 "")

[ghost]

the compression methods are for chd only CHD_CODEC_CD_LZMA CHD_CODEC_CD_FLAC CHD_CODEC_CD_ZLIB

so hdd based chd (or whatever other format chd has) was probably not included in the making of this chd port. probably can do check for chd->codecintf[i]->compression if its valid for current codebase or what not...

[orbea]

If this was fixed it might be easier to debug.

https://github.com/rtissera/libchdr/issues/7

[i30817]

Reminder that this is still happening and it's still a major scanner usability bug, more than one year after the report.

[DewmBot]

I as well am having this issue. Is any more logs needed for this... or? Scanning single file or directory has the same issue. Scanning through Desktop UI does not work either.

[mirrornoir]

Just wanted to speak up that I've encountered this bug as well. CHDs for other consoles work fine but MAME CHDs cause RetroArch to crash when the playlist scanner encounters them. If the games are manually added or run, they work fine. As soon as you start including CHDs in your MAME library you effectively loose the ability to use the playlist scanner.

I ran into this issue in RetroArch 1.7.7 running in Arch Linux. I decided to play Beatmania THE FINAL MAME/bmfinal.zip and once I ran the playlist scanner, RetroArch crashes to desktop the instant it finds bmfinal's hard disk CHD MAME/bmfinal/c01jaa11.chd. Just scanning the bmfinal directory that contains the CHD is enough to crash RetroArch. I can provide logs or any other troubleshooting if needed.

[i30817]

No one maintains the scanner, which is why it's on this terrible state and 2+ years old bugs keep on trucking. Frankly, no one should be using it and code quality in retroarch would increase if it was ripped out into its own subprocess and reimplemented with better strategies for classification. Lol, decompressing chds to classify games.

I had some hope that exactly that was going to happen, because the libretro database tool the scanner uses can be built standalone, but it seems the effort petered out.

[DewmBot]

Damn, that's a real shame. The problem I'm having is I can't actually launch the game through launchbox unless it has been added properly through retroarch.

[i30817]

Damn, that's a real shame. The problem I'm having is I can't actually launch the game through launchbox unless it has been added properly through retroarch.

MAME collections can be scanned if you move the subdirs (holding the chd's) out and move them back in after the scan. You see, retroarch doesn't actually need to scan the chd's for MAME because the mame dats only need the main game archive to recognize it. Ridiculous and ugly, but there you have it. You only need to scan the chds if you want to use the MAME romset with another emulator core (which tbf, you might want to).

The rest of this post is a rant, and if you don't care, just ignore it I guess.

This problem is caused more because the code handling the chd's misses a case where the chd is not a cd then crashes because it's attempting to decompress a hard drive image as a cd image, and because retroarch likes to pretend all kinds of roms are 'agnostic' to the emulator or system that runs them, which naturally gets fucked up when the format is shared by many consoles or the container format is multi-type and the many simplifications and hacks RA does to avoid having user intervention on the scanner fall apart.

It already doesn't work with translations and hacks of cd games because the crc32 is never calculated actually, even if the checksum is on the database. Which let me tell you, as a database hack checksum PR contributor is highly discouraging (not to mention they don't want to add a actually up to date and with tool assisted update checking, NES and SNES hack database because it's 'too big' lol).

I actually shouldn't be too hard on them right now, there is good movement on updating the playlist format to json with a open pr by a newcomer, which should give more flexibility to important parts of this code, and it's legitimately a complex problem to balance speed, reliability, memorization, user intervention, the fact that dumping groups are incompetent and that cd formats have so many useless variations and no common checksum (except chd, which no dumping group uses, except MAME, which coded it from the ground up, see 'incompetent') and then even if you find the checksum, it doesn't often correspond to the file you have to feed the emulator (cue - launcher file - vs bin - actual executable track with the right crc32).

That last notion is completely 'ad-hoc' inside the retroarch scanner, and basically hardcoded with some fallible 'rom platform detection'. Some cores need to scan track 1, some need track 2. Some cores need a index file, some don't. Some users want to run dodgy closed source programs (like alchohol 120%) format files, some realize that is insane on a open source system, etc. I have some open suggestions to make the scanner usable by people who care, but to be fair, they'd be difficult to implement in python, much less portable C, so i can see why no one comments on the issues.

[rtissera]

Hello,

Fixed https://github.com/rtissera/libchdr/issues/7 Meanwhile I do confirm HDD-CHD codecs are not supported.

The original libchdr stuff was done to support CD-based CHD files. It can be expanded to support other use cases but this requires rewriting a lot of C++ code back to C from MAME source. Also if my memories are right, I gave up on this because of too much code tied to MAME core.

Might have a look again if it makes sense.

[Jamiras]

While testing CHD support for PSX hashing I encountered a null reference and added a null check: https://github.com/libretro/RetroArch/pull/9405/files#diff-8bb4632333237a5271ba90ec22b608d3R1040

This change was in chd_close, which corresponds to the above stacktrace.

#0  0x0000555555bf53fa in chd_close (chd=0x5555577a0020)
    at libretro-common/formats/libchdr/libchdr_chd.c:1030
...

I have no idea if it fully fixes the originally described issue.

[rtissera]

@Jamiras indeed, that's a decent fix for missing codecs. EDIT : I have similar upstream fix, probably libretro-common did not pulled it recently.

[sonninnos]

CHDs are no longer getting scanned with current info files, so closing time.