M3U files required by cd-based beetle cores but not supported well at all by UI and features available for CUE scans

[birdybro]

Description

Multiple problems with M3U files in the user interface as of version 19.0 (and as long as beetle cores have required m3u for multi-disc on-the-fly changing). M3U files are a requirement for using beetle cores that are for CD games with multi-disc support. Here's the problems that need to be addressed as it's very unintuitive to use M3U's but still required:

• Scanning directory by default does not scan in .m3u files, this has to be done manually. This doesn't make sense if it's required to use them for basic functionality.
• .m3u files are not checked for in the game information database (e.g. Chrono cross cue files get thumbnails and database information, m3u gets no thumbnails or information, just the title)
• If a game has an .m3u file being utilized (meaning it's a multi-disc game since this is necessary), the scanner should ignore the cue files of the same name/database entry, but currently only gets cue files, and as said before ignores .m3u. It ought to include only .m3u files for games that have them, and cue files for games that don't have an .m3u, to not import lots of unnecessary games. manual scanning does not help with this problem as you will get either only .m3u (which most games don't require), or both cue and m3u, or you exclude m3u as your only 3 options. That or you manually edit the playlist file laboriously if you have a large collection.

Expected behavior

I expect to "scan directory" the PSX directory and have my .m3u supported games have a thumbnail, game information from teh database, and my cue-only games to be scanned in like before, with the cue's for m3u-supported games being excluded from the list.

Actual behavior

Scan directory adds only cue files from the psx directory, ignores m3u, and if you manual scan the directory for .m3u, then it still doesn't use the games database and therefore gets no game information or thumbnails for the m3u entries. If you scan manually for m3u only, you are excluding cue, if you scan manually for cue, you miss m3u, and if you scan manually for both, you have excess cue entries from .m3u enabled games.

Steps to reproduce the bug

Import --> scan directory --> a psx folder with a mixed set of m3u-using games and cue-using games --> look at the psx playlist and see no m3u files --> manual scan for m3u --> update thumbnails --> m3u games are now added, but no game information and no thumbnails and redundant cue files for each disc have to be manually removed

Version/Commit

You can find this information under Information/System Information

• RetroArch: v1.9.0

Environment information

• OS: Windows 10 Home Feature Update 2004

[i30817]

1. It actually can be done automatically, the RA project is just resistant to doing so. For instance i implemented this for linux a while ago. And i'm far from the only one. It's understandable something like this isn't in the 'automatic' scanner but i wouldn't say no to someone contributing it on the manual one, since typically only some cores can even use m3u.
2. database problem. There is a technical solution but i requires discipline from contributors, so it wasn't done, badum tish.
3. The script above and manual scanner with exclusion of cue solved it for me.

[birdybro]

1. You might be confusing this with the previous issue I raised with an automatic m3u generator feature request, which we both talked about back then too. :) - This is in regards to the scan directory behavior not being appropriate for the most used core when it comes to multi-disc games.

2. How much technical knowledge is it to do this? Is it simple data entry for the most part? If it is, then I guess I could start knocking them out.

3. This doesn't generate a playlist (.lpl) file though. I remember when we chatted about this last in my feature request here previously that the idea was to integrate it.

My issue raised here assumes that the user has already made their own .m3u files, and the scanning of the directory doesn't work as it ought to. I think I will spin off your gist like i had previously planned to since I have gotten slightly more experience messing with code since we last chatted about it, just as a side thing.

[i30817]

Guess i forgot. About 2, it's a scripting problem and a 'enforce contributors contribute files in a specific way' problem.

edit: had a better idea than symlinks in step 7. edit2: read a bit about git and extended attributes wouldn't work but symlinks do.

The idea is simple. Someone contributes a image with listing the game console that applies, and the file(s) with a filename like Xenogears (USA).png - xenogears the game has two discs, but assume the image is the same for now. If they want a particular edition or cd to be different they can be more specific, aka 2 files, 'Xenogears (USA) (Disk 1).png' and 'Xenogears (USA) (Disk 2).png'

Now for the script. Periodically the person updating the thumbnail database runs a script that:

1. takes the console name the file corresponds (easily done by seeing the parent dir in the libretro database scheme for single files committed) - this is the reason it isn't actually necessary to ask the contributors platform, they already did when they placed the file in a dir.
2. With the png filenames, remove the extension and a very simple parsing operation for the rest: text that is without groups of parenthesis around it, is placed as individual entries on a text set. Save the groups in a set for each one. Why remove the groups? Because some dumping groups have nonstandard ideas and can easily place extra groups in between what you expected or change the order. But save the groups. Why keep the text separate? To do two+ searches if necessary to keep the specificity (should be uncommon).
3. Anyway, you're not done! Take the groups you took, and for three types in particular, multiply them. '(Rev. 1)' becomes both that and '(1.1)', (EUR) becomes both that and (Europe), '(Disc 1)' becomes both that and '(Disc 1 of #game_discs)' - all the variations for this of the common dumping groups at least. This is easy too, except the '#game_discs' part, which is possible as you'll see next.
4. search on the specific libretro console rdb for names where for the text set consider the set as one in where all must match (and maybe match from after the last index of the last match, so this 'set' here should be a list); Now if there was any 'cd group' in 3., you find out the number of max cds in the first of rdb results you got from the rdb and fill that out in the 'cd group set', do so; now consider the sets holding groups as sets where 'any' could match (this enforces the 'specificity'), filter again the results.
5. You have listing of all the games that primordial .png applies to in the libretro database now and could copy the pngs. Obviously this only applies to names in the database which might be considered a disadvantage, but at least you also get metadata that way. The files also include hacks which often add a [hack metadata] group at the end of the file and are in the database.
6. Remember back in step 1 when i said you could have 'more specific' names? Yeah, if the file already exists, don't replace it until the admin or a commit removes it. Obviously, you want sort by name size from largest to smaller on step 1, so the more specific names go first if the contributor is doing them at the same time.
7. Regenerating everything: While this idea is ok for running this for single files once, you want to run this on a script, many times and with previous output in the same dir as the input. So you should think of a way to figure out to distinguish files copied by the script and files manually inserted, and skip the produced ones. Also if you want to regenerate everything completely, just delete all the files with that 'mark'. Various marks are possible: but they suck. Extended attribute would be ideal but git doesn't preserve it (it only does the executable bit and the symlink bit). Edit: meanwhile i found that libretro-thumbnails already uses symlinks there so using those is quick and easy as long as they point to the older names correctly. They even adopted the convention i had above for the 'base.png name', so yea, it seems like it would work.

Finally: What about the M3U?. The above 'created' the missing entries on the database, as much as possible. But while you could make RA use Xenogears (USA).m3u as the standard name and leave it at that, users are crazy, and probably want to rename the M3U (add a abbreviated hack name that isn't upstream but the full hack name is, etc). Make the scanner assign the image of the first entry in the M3U for M3U's is my personal preference to work around these issues. This is a separate modification to the script above though.

[i30817]

edited the algorithm above to hopefully be a bit more clear (edit: again). Anyway, i find it doable, certainly in python or whatever libretro database uses, and with a standalone rdb search tool that retroarch can compile. It would balloon the size of the repository though i'm not sure they care. Users with full sets would probably appreciate if it was possible for the RA thumbnail downloader to use symbolic links for these - for multidisks, and multifloppies, since some floppy games can go to 10 or more and 3-5 is not unusual.

It's better to encourage users to use M3U than make the thumbnails have a complicated symlink system thats not portable, and push cores to support M3U to make users save on thumbnail bandwidth. A project like Ludo could even enforce that the only format their playlists support is M3U and if they could have core support, profit by having a simpler UI and experience as well as a less huge thumbnail folder.

But who knows, maybe some github guy make something simpler like a github plugin that for commits for libretro-database and runs a dialog 'Are you sure you don't also want to add this list of files as aliases for your 'Game.png'' (the list being given by those autocorrect word algorithms) and copies into filename check the committed image.

[i30817]

Btw @birdybro , i remember @jdgleaver implementing something like the filtering for files inside m3u scanned that you mention doesn't work? Am i just imagining it @jdgleaver?

[jdgleaver]

Yes, if you scan an M3U file with the manual scanner, it will automatically exclude any files that are referenced inside. Also, if you clean a playlist (Manage Playlists > Clean Playlist), it will remove any individually-listed items that are duplicated inside M3Us.

[birdybro]

I must have been doing something wrong with the manual scanner, I have confirmed that it works, but that doesn't address the other issues. Can't this same behavior from the manual scanner be imported into the automatic scanner?

[birdybro]

As far as .m3u generation, I found a great tool for this which I believe works cross-platform, maybe the method could be implemented.

https://github.com/tralph3/Cue-Maker

Or maybe this person would be interested in contributing to retroarch with the m3u generation python script they made @tralph3

[tralph3]

I actually made the script thinking about retroarch, but I don't think retroarch should be in charge of making m3u or cue files. I do want it to detect the m3u automatically and ignore the cue files inside it so I have a single entry with multi disc games (this is currently possible, but I have to manually remove the cues and add the m3u with the desktop app).