Scanning in Archived/Nested Sega CD games returns nil

[klepp0906]

First and foremost consider this:

• Only RetroArch bugs should be filed here. Not core bugs or game bugs
• This is not a forum or a help section, this is strictly developer oriented

Description

Boy - I'm keeping you guys busy with the complaints eh? I'd imagine this one extends to any system archived in a similar fashion, however I'm going one by one and am currently working Sega CD.

Hierarchy is archive.zip > folder > bin/cue insider folder.

After RA is done scanning, no playlist is created.

Expected behavior

RA to create a playlist, same as with any other directory/structure.

Actual behavior

No playlist is created, I surmise due to not looking inside sub-directories? Or maybe it has something to do with having multiple files at the end (bin/cue?) and not knowing which to peg? (unlikely id imagine)

Steps to reproduce the bug

1. Create an archive with a similar structure.
2. Attempt to scan the directory/file.
3. note RA not picking it up.

Bisect Results

I'd imagine has always been the case, but can't say for sure.

Version/Commit

You can find this information under Information/System Information

• RetroArch: 1.7.6 build date apr 24 2019

Environment information

Win 10 Pro

[andres-asm]

What's the point? RA can't load this kind of content

On Fri, Apr 26, 2019, 05:21 klepp0906 notifications@github.com wrote:

First and foremost consider this:

• Only RetroArch bugs should be filed here. Not core bugs or game bugs
• This is not a forum or a help section, this is strictly developer oriented

Description

Boy - I'm keeping you guys busy with the complaints eh? I'd imagine this one extends to any system archived in a similar fashion, however I'm going one by one and am currently working Sega CD.

Hierarchy is archive.zip > folder > bin/cue insider folder.

After RA is done scanning, no playlist is created. Expected behavior

RA to create a playlist, same as with any other directory/structure. Actual behavior

No playlist is created, I surmise due to not looking inside sub-directories? Or maybe it has something to do with having multiple files at the end (bin/cue?) and not knowing which to peg? (unlikely id imagine) Steps to reproduce the bug

1. Create an archive with a similar structure.
2. Attempt to scan the directory/file.
3. note RA not picking it up.

Bisect Results

I'd imagine has always been the case, but can't say for sure. Version/Commit

You can find this information under Information/System Information

• RetroArch: 1.7.6 build date apr 24 2019

Environment information

Win 10 Pro

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/libretro/RetroArch/issues/8642, or mute the thread https://github.com/notifications/unsubscribe-auth/AANEFUCNILOF62KHEPZ3X23PSLJS7ANCNFSM4HIUMW6A .

[klepp0906]

odd - loads just fine for me. always has

[orbea]

What's the point? RA can't load this kind of content

I've seen it both work and not work (Depending on core?), but its kind of not worth pursuing since its almost user error. Compressing bin/cue files with zip does not save space and is very slow to decompress.

[klepp0906]

yea i use genesis plus gx, but have also used picodrive. As for it not saving space - this I was not aware. Never thought to check. Definitely not slow to uncompress, at least on my machine - but if its not saving any space or even marginal space - thats irrelevant.

I I have to ask then - assuming I decompressed all the archives, it would leave me with individual folders in which a bin/cue resides. I imagine retroarch can scan the individual folders without having to scan them "individually" if that makes sense? Aka i can do the entire directory?

EDIT - yea definitely saves space. tested a few archives. 20-25% I cant begin to guess how much work, or otherwise this would take to implement or address - but i imagine its a feature that wouldnt be constrained to just sega cd. Not to mention how many people compress what they can to save space. (aka id imagine it would serve a good number of folks if it were available)

that being said - your the kings of this castle :)

[orbea]

Sorry, I was thinking about larger files like ps1 games and not genesix plus gc or picodrive which my concerns might be completely false for...

I am also not the king of the castle, I view this as a community project and am just trying to add the discussion. :)

[klepp0906]

"kings" plural! I know the size and scope of this project takes a slew of you guys. Your definitely active and helpful thus far though. Ive only began posting as of late but ive been combing through RA pretty thoroughly and will be doing so for the forseeable future so the likelihood of more posts is .. well . likely :p So far youve been in every one I think lol.

Back on topic though - yea definitely saves a meaningful chunk of space, and RA uncompresses them on the fly with a couple seconds at most to run. Hopefully someone will take the time to look into it - of course im bias'd but i imagine having it take nesting into account will certainly help "big-picture" wise and futureproof it a bit more. Not in the sense that future games will need compression of course, but that people will put it to use going forward across the board I imagine.

[andres-asm]

It can't possibly work for bin/cue and even less so for multi bin/cue. RA only extracts the LOADED FILE.

The only core that supports compressed sets is Kronos, and that's because the core itself supports it (so zip is a supported extension).

Also, it would certainly be a lot slower than just loading. Depends on the dataset of course but extracting a big game will take a while compared to just loading.

[klepp0906]

yet you have someone else in here saying it does depending on core. I thought it may be due to steam shortcut (even though thats unlikely to affect) so i tried via simply navigating within retroarch and yep, still works.

cant speak for multi bin/cue but single bin/cue works just fine.

im on a nightly build if that makes any difference, always have been afaik so if the non nightlies treat it any differently.

I mean, i can take a video if seeing is believing :P

again we're speaking in regard to Sega CD exclusively. ive tested with gen+gx and pico, the others i cant say.

[i30817]

Support for chd on more cores would be great, but only if it's actual streaming and not 'extract to tmp then feed to core'.

It's kind of understandable that the above isn't common, since cd-mounting for consoles is part of hardware emulation, so it's a difficult feat to externalize that, and RA probably won't so we'll keep only getting chd or compressed support that doesn't imply a massive disk write only on some emulators. Some emulators support virtual cds like cdemu, but none on RA that i'm familiar with (and besides, all virtual emulators i know won't support exotic 'cd formats' like the dreamcast one because it would require emulating a completely different drive).

I'm hopeful that code from mame cd/dvd emulation (that will naturally support streaming chd) will spread downstream. Hopeful but not optimistic.

The non-support for 'streaming' compressed archive support in RA is the main reason i use filesystem compression instead of archives. This is also a pain, because RA only extracts internal checksums from zips, and i'd much prefer crc32 scanning, but since RA currently uses serials that slowness is not currently a problem.

[andres-asm]

@klepp0906 it can only work if you loaded the bin directly, not if you load the cue.

[klepp0906]

@fr500 perhaps thats what its doing under the hood then. I know my glob for the parser that adds shortcuts to steam simply points to the zip.

same with RA, i just navigate to: and select load archive. Perhaps if it were multiple within the archive it would fail, otherwise it defaults to the bin and thus works, at least on all the games ive tried and the core im using.

[i30817]

Loading a bin fails to load cd audio if the game has tracks in separate files... redump does this.

So i'm extremely surprised that there is any cd platform where RA doesn't accept a cue when given.

[andres-asm]

If you go and "load archive" in case of a zip or 7z that contains bin + cue files it will just extract the first file in the archive and load it, that's all that's happening.

So yeah if it's a single bin it will work IF the core is able to load that single bin normally Nothing else will.

And yes if it's a multi-bin image like on redump sets:

There will be no audio. The reason this is working is pure luck.

In my opinion this should be disallowed until properly supported (IE: if the user loads an archive, and core doesn't support zip or 7z directly in the supported extensions list, it should bail with an error)

The fact that it's working at all is an accident and bound to break in several use cases.

[andres-asm]

@twinaphex ok there is a potential issue here, and it's something that users may have been suffering for a while.

While what I'm about to suggest is not a solution, it will at least prevent weird issues and even crashes in some cases.

So as you are aware, load content is complex, particularly because we have this "suggested cores" system (which is neat)

The user workflow is most likely Load Content->Select Core from Suggestions

So this poses a few challenges:

Scenario a: file is not an archive --> Solution: Show suggested cores

Scenario b: file is an archive --> Solution: Expose two ways to load content, browse the archive, load the whole thing ------> Browse archive: -----------> This one is mostly good, it will load extract exactly one file from within the archive and load it ------> Load archive: -----------> This one is a bit fucked up. I'll elaborate below

Scenario b1: archive has one file, that file is supported by several cores --> In this case RA will show you the cores that support the internal file AND the cores that support the archive format.

What should happen instead in 1 file per archive is only show the cores that support the internal file, so despite using browse filter based on the INTERNAL EXTENSION

Scenario b2: archive has several files

What to do? only present suggestions based on the ARCHIVE EXTENSION because we can't really guess which file should be loaded. It would only show MAME, FBA, KRONOS, the user would be unable to go forward but it wouldn't do bad stuff either

Bottom line:

1. only filter on the internal extension for 1:1 archives
2. only filter on the archive extensions for multi file archives

[andres-asm]

@klepp0906 sorry for derailing the issue this much 🤣

[klepp0906]

not derailed at all :) glad to at least see people giving it some thought ;p Hell im even learning something lol.

awesome post btw ^^

[orbea]

It would only show MAME, FBA, KRONOS, the user would be unable to go forward but it wouldn't do bad stuff either

What if they do continue forward? Are all these cores capable of handling wrong content without blowing up? I do agree with your suggestions.