[Feature Request] Automated m3u generation toggle and script for Import --> manual scan

[birdybro]

Description

I would ideally fork and do this myself, but I'm a total amateur and it's not reliable for me to complete it or start it for various reasons (but it's directly related to something I want to do for my first real program I want to start making), so might as well convey the concept here to see if anyone is interested in doing it, who can. Thanks for reading in advance.

My request is to have a feature added to the import --> manual scan section, an on/off toggle named something like "Automatic .m3u generation". This toggled option would call a script (upon scanning the content directory being initiated) that would enumerate the targeted import path's filenames, and generate .m3u's for every .cue+iso/bin in the folder in a smart way that would result in the m3u's being generated automatically without the user having to laboriously create .m3u's for every single game (which can be a problem if they have hundreds of multi-disc games to go through). Then the script (or maybe better for the toggle rather) would finish by signaling to whatever other programs run to import the scanned files to a playlist, that they should only import the .m3u's if they have allowed Retroarch to ignore unknown filetypes in their settings. It also might be worthwhile to place these automatically generated .m3u's in a retroarch subfolder named %APPDATA%\retroarch\m3u for windows or ~/.config/retroarch/m3u for linux, or maybe that's not correct or not worth it on second thought.

Expected behavior

Ideally I would expect an ignorant user's first time running through a game like Valkyrie Profile (to use my example) from a CUE/ISO 2-disc format, to play the game, and when it says to change disc, they toggle the quick menu, and change the disc or they select the second disk file, and all is well, they continue on with the game. Or if they need a .m3u file, the program would make it for them in some way so they wouldn't have to make 100s of them manually for a large library they may have.

Actual behavior

Mostly a list of necessary consequences to the user impacted by current design: The user does their playthrough, and hits a wall with the change disk event. they then have to close the game, losing progress potentially, could be hours, because their .mrc files, .srm files, and save state files are also not going to be named the same as the .m3u file if done accurately. They then have to manually create an .m3u file after reading mostly forum topics about it they googled, most of which are outdated with regards to the quick menu options. they then have to rename the memory card, save files, and save state files to match the .m3u filename they have generated to get anywhere.

Version/Commit

You can find this information under Information/System Information

• RetroArch: 1.8.5 Windows Binary (just saw 1.8.6 came out when writing this, didn't notice anything in CHANGES.md that I think would be doing what I'm suggesting)

Environment information

• OS: Windows 7 x64 Home Edition (and reproduced on Windows 10 x64 Home Edition)

[i30817]

If you were in Linux, there is a easy solution to this:

https://gist.github.com/i30817/ba37fbb2b3c6e34ff926ad833f465055

but as you are not, I dunno, you could try to port the script to python? Shouldn't be too hard with a example.

[i30817]

Found this just by searching github for 'm3u retroarch': https://github.com/stevesolomon/retroarch_m3u_generator

Looks like a much more primitive version of my program that will kinda sorta work if there are no disks above 9 or you don't have 2 releases (1.0 and 1.1) of the same game in the same dir or you don't have a cue with a iso inside or if you don't care about the name getting false positives from opening brackets before the (Disc ...) part or if you only want relative paths or if you don't want the option to name the cue the same as the first disk (to get images from disk 1 in retroarch if the game is right).

And a perl program that i'm going to assume is useless on windows. edit: i assumed wrong, the program has a compiled exec, and the interesting feature of creating a playlist. https://github.com/pkos/m3umap

[birdybro]

This was my plan originally, to start learning more about coding by making a script in python using some examples I found these originally when looking:

https://gist.github.com/shidarin/4c8d07508ff1fdd075a3 https://github.com/alondero/m3u-generator

In addition to pkos/m3umap that you linked above...

But I would probably be unable to get this from simple renaming to something that could be integrated into retroarch the way this feature suggestion is proposing.

The toggle and linking to other toggles in the import scan are newer ideas I thought of as I planned my script and looked around so figured I'd share.

[i30817]

M3U generation has been suggested before and ignored, which is why i created the script and i'm directing you to alternatives.

My idea about this is that there is no actual need for 'real' files, just the playlist is enough to create single games «m3u» (with the same method my script uses) in the playlist. It could be indeed a switch (though it'd probably not work well with music or more exotic 'roms' that don't have the dumping groups naming scheme).

However, much like the rest of retroarch, the playlist is C code and that is not the most friendly of places to use abstractions like this (combining 'real files to feed to the emulator' with 'virtual m3us to feed the emulator'). You're braver than me if you considered coding this.

[birdybro]

Thankfully i was told by one of the contributors on discord that retroarch does have some scripts that aren't in C/C++ which are called for things in the main interface. They said that retroarch has one python script it uses already. So doing it in C/C++ may not be necessary. I really appreciate the insight and feedback!

[i30817]

BTW, if you go for this i want to call your attention to a problem that other implementations of this i noticed screw up.

Basically, if you go the route of allowing any amount of games in the current directory being processed (and you can't know in advance what the user did), you can end up with the situation where you have two versions of the same game in the same dir:

FF VII (Disc 1) (1.0).cue
FF VII (Disc 2) (1.0).cue
FF VII (Disc 1) (1.1).cue
FF VII (Disc 2) (1.1).cue

The problem here is that string sorts go left to right during matching, so even if you do the obvious thing of distinguish games through 'name-(disc group)' you can easily end up with

FF VII (1.0).m3u:

FF VII (Disc 1) (1.0).cue
FF VII (Disc 1) (1.1).cue
FF VII (Disc 2) (1.0).cue

FF VII (1.1).m3u:

FF VII (Disc 2) (1.1).cue

The hack i used to 'solve' this, and since bash sort doesn't allow to give a comparator function:

1. add a 'new' string suffix to each to be sorted entry of the name, with the disc|disk|floppy group moved to the end, separated by \0 (illegal character in filenames) FF VII (Disc 2) (1.1).cue\0FF VII (1.1).cue(Disc 2)
2. sort based on that part of the name (bash sort does have a -delimiter and a -key switches), with a 'version' sort (so 10 comes after 2 not before).
3. extract only the first part after everything is sorted, and 'change m3u' when the current name - the 'disc group' doesn't match the first name - disc group of the current 'm3u'.

You probably have a much better way to do the same in python but these bugs should be avoided anyway, the sort of different versions of the same game and of games with more than 9 'roms' need to be correct.

[birdybro]

This is a really good thing to consider. I was thinking about it the other day in another context, if one is misnamed. I guess that's on the user for naming their files wrong. But your example is a more realistic best case scenario on the user's part. This is very helpful.

I will add, that the only major difference I can see (functionally in this case with what you were referring to) between bash (what you used) and python (what I will likely use) is bash treats outputs as strings and python treats outputs as objects. But in this case, since they are filenames, the exclusions and procedural entries you are referring to would be mostly using strings in the script anyways, so this difference would be moot. That's my assumption at least.

[i30817]

I edited the message to explain how i know when to change m3u to a new one. It's easy if you just remove the '(Disc ###)' from the filename after everything is sorted and compare the first in the m3u to the current. If they're different, it's a new game and needs a new m3u.

This even works for redump compilations, because they name the different games differently instead of just placing 'Disc 4' or something. For instance Metal Gear Solid and Metal Gear Solid VR Missions, which came in the same package have different names instead of VR Missions being disc 2 of Metal Gear Solid.

As for the name of the M3U, it's more beatiful to use Final Fantasy VII (1.0).m3u but i added a compatibility switch to use Final Fantasy VII (Disc 1) (1.0).m3u because Retroarch doesn't behave well at all with images otherwise.

Heck it doesn't behave well even with the names of the first disc. There are a lot of images missing even on the ps1 collection, much less the others. The last option i did was to make the paths of the m3u relative to the location of the m3u or absolute. It's the kind of thing where you hope that can be portable in the future but really isn't because OSes like windows and linux have different path separators. There is a open PR that tries to handle that but doesn't touch m3u - and it probably won't, since m3u are passed to cores directly IIRC.

I could code another switch to dump the m3u in the same directory as the cues, but decided not to, since it's simpler to delete in a directory apart imo.

[birdybro]

As for the name of the M3U, it's more beatiful to use Final Fantasy VII (1.0).m3u but i added a compatibility switch to use Final Fantasy VII (Disc 1) (1.0).m3u because Retroarch doesn't behave well at all with images otherwise.

This must have been a problem with older versions of Retroarch. The only problem I ran into was I had to rename memory card, saveram, and savestate files to match the .m3u I had created, since I started using the Game (Disc 1).cue file and made an .m3u and am now using Game.m3u for instance. And I only had to do that since I wanted to use the .mcr/.srm files primarily. Those are named the same as the filename of the game you are playing, basically.

[birdybro]

I had to help a guy at work with pathing similar to what you were mentioning, but he was using Python. They have a new module called pathlib that seemed to clear it up for him. You mentioning that kinda reminded me and is helpful as well. Thanks.

EDIT: https://medium.com/@ageitgey/python-3-quick-tip-the-easy-way-to-deal-with-file-paths-on-windows-mac-and-linux-11a072b58d5f - pathlib article I sent him for reference, partially to remind myself here as well.

[i30817]

It's not exactly a problem with retroarch. It's a problem with the database, which seems to sometimes in multidisk games either have the TOSEC kind of naming (Disc 1 of 4) or one 'beautiful name' Final Fantasy VII (USA).

Not all games have the 'beautiful name' yet, but apparently when they are, the redump type (Disc 1) is deleted (or never existed?). Maybe the thumbnail downloader is removing the Disc Groups already (to save database space) but not all games are converted to the short name?

For instance write 'Final Fantasy VII' here: https://github.com/libretro-thumbnails/Sony_-_PlayStation/find/master

edit: if you type '(Disc 1)' you can see the few that still use redump type names. The very few. Also type '[T-En' to see two random translations with screenshots just for them.

[i30817]

Pathlib is interesting. The open PR trying to make paths compatible between windows and linux can't use it, but maybe i should link it as prior art.

[birdybro]

Also, worth considering, apparently as of Retroarch 1.8.6, according to the update:

https://www.libretro.com/index.php/retroarch-1-8-6-released/

(Side note: 1.8.6 also adds a simple but feature complete M3U handling library – this may have additional use if someone wants to add the ability to generate M3U files for existing content…)

So a good amount of the legwork to making this done on existing playlists, or at least the ways to connect it to a script like this, is already there.

[i30817]

Ok i've commited some images to the thumbnails psx repo and i noticed some things that cause problems with m3us.

First only some multidisk games have the 'base form' without the (Disc), which is obvious. Second all thumbnails are there without the version number if it exists. Third, it's really random what gets committed. Many '(Disc 1 of X)', some '(Disc 1)', some 'base forms', some have extra '()' groups for studio etc. Fourth, sometimes it's just a outdated redump naming. Either me outdated or the repository outdated, not sure.

I opened this in the hope this jungle can be tamed - not sure it can, if RA is not using git directly https://github.com/libretro-thumbnails/Sony_-_PlayStation/issues/100