[Feature] Improve Arcade ROM importing experience

[TylerVigario]

Oxyromon doesn't show what folder or archive the file it's trying to match to it's nearly impossible to import Arcade ROMs at the moment.

It would be awesome if oxyromon would have some method to automatically select the ROM a file matches to. Since my ROMs have been validated outside of oxyromon it makes sense to me to use the folder/archive names.

[alucryd]

Yeah, every time I match an arcade game it reminds me I need to add that. It's also on the roadmap, but it seems I haven't explicitely listed it on the todo yet. I should have some time over the weekend to tackle some, if not most, of the issues you've opened.

[alucryd]

https://github.com/alucryd/oxyromon/commit/e0428b1f6f2fc7aee8fd7868d86f7801f421516d should be a huge step forward. It will now use the archive name to automatically select the right game (if any) and in case of merged archives, it will also use the name of the subfolder inside the archive. Also printing the name of the archive now when asking to select roms in case there are multiple matches.

[alucryd]

And one step further: https://github.com/alucryd/oxyromon/commit/c84b14af296431ab6dd007a4c1557821693cddab

Also using the file name now, some games can contain 2 identical files with different names, this one will avoid presenting a prompt in such cases and automatically choose the matching name. Also added a None option when choosing between multiple matching ROMs, there's almost 100% chance with arcade that when you're presented with a prompt you will want to select it after these changes.

Imported an older set from my mister, I think it's based on MAME 0.238, with both the latest FBNeo DAT and the MAME 0.251 DAT. Had to discard some things due to the older set, but overall it went pretty fine and I could just spam the space bar to select None. I'm compressing them games back to zip and will revisit the rebuild-roms command again, hopefully it still mostly works.

Oh, and the completeness should be fixed. Didn't notice any discrepancy in the web UI, and now that 1G1R computation is changed, I made its filter button separate from the complete/incomplete/ignored set of filters. 1G1R games also appear bold now instead of blue, as a game can very much be both incomplete and 1G1R now if strict mode is used.

I was afraid the sqlite database would crumble under the load of a MAME DAT, but it's still holding up amazingly well, I'm more and more impressed with sqlite every day!

Edit: Looks like rebuild-roms is still working fine, although it's abysmally long and still running after a good half an hour going from split to non-merged, because zip is not multithreaded, and my server doesn't have the strongest single core performance out there. There is a native rust implementation for zip I could try instead of calling 7z, it should help a lot because it would allow me to stream data in memory instead of going through temporary files. It doesn't have all the features 7z has though, like deleting or renaming a file in an archive so I would still need 7z for some things. I could also try some parallelization with rayon, although that would completely mess up the CLI output as it is now, will need to try the fancy multi-progress from indicatif, I've always wanted to take it for a spin!

[TylerVigario]

@alucryd I noticed 1G1R mode doesn't work for Arcade DATs, specifically MAME 0.251. sort-roms puts no ROMs in the 1G1R folder. Maybe I am doing something wrong?

[alucryd]

It's not implemented yet, and is very very very low on the todo list. For consoles the regions are extracted from the game names with shiratsu-naming, using the no-intro convention. It's a great system and very easy to implement.

MAME doesn't use any convention, it's utter chaos with names seemingly dating back to the old IBM DB2 databases with an 8 characters limit... Some of the descriptions can be used, but not all of games have their region in it (and by that I mean many don't), and then there's stuff like US, USA, USA followed by a random amount of digits and what have you, and they don't use the release tag from the Logiqx specification either.

In the current state of arcade DATs, using any kind of 1G1R on them would do a very poor job and poor jobs are not my cup of tea... (do I sound salty? I think I sound salty xD)

[TylerVigario]

In the current state of arcade DATs, using any kind of 1G1R on them would do a very poor job and poor jobs are not my cup of tea... (do I sound salty? I think I sound salty xD)

I have found multiple implementations of 1G1R with these DATs that works pretty well. Retool being my favorite, but CLRMamePro & RomCenter also work pretty well. Would have to rely on P/C data soley I imagine.

EDIT: Glad to know I am not doing anything wrong. Can't wait to see this implemented in the future!

[alucryd]

@TylerVigario Sure, disregarding any region information I could easily separate parent from clones, it might even work fine in split mode, but doing 1g1r with arcade and regions essentially locks you into non-merged, I'd even recommend full non-merged to be sure you're not missing a bios somewhere. Merged is also out of the picture since parent and clones would sit in the same archive so that defeat 1g1r's entire purpose.

In any case, finally have some time to look at making rebuild-roms actually usable when going split -> anything, it ended up taking upwards of 2 hours yesterday, that's bad, that's very very bad :/

[TylerVigario]

@alucryd That is a good point! I run fully non-merged for my Arcade ROMs to remove unwanted/unneeded files anyway, which is mostly the point of 1G1R-only mode.

In any case, finally have some time to look at making rebuild-roms actually usable when going split -> anything, it ended up taking upwards of 2 hours yesterday, that's bad, that's very very bad :/

Glad to hear that maybe some parallelization will make its way into the slower operations!

EDIT: Forgot to state that the new Arcade import experience is phenomenal, great work!

[alucryd]

Oh well I got distracted: https://github.com/alucryd/oxyromon/commit/99df462de38ca724c9cd404fb982ca9a9c8c428c

import-roms should now automatically find the correct system without prompting the user. It is still possible to force the prompt by passing -s and I recommend doing so for arcade to use the new autodetection by name, but consoles should love the new mode. I will probably extend the name autodetection to this new mode later though.

It will still prompt if a JB folder is detected mind you, will see if I can make that one smarter later.

[alucryd]

No need for parallelization after all!

https://github.com/alucryd/oxyromon/commit/d124a6afb94f69bfd3fc80ae0f75f85d2559beba

Reworked some of the logic of rebuild-roms, and used the native zip support in place of calling 7z where possible. It's so blazingly fast there's simply no contest, went from several hours to under a minute on my test sample xD

Now I wish the native zip crate had support for more operations but it's quite limited and I still need to call 7z for a few things (like renaming files in archives, or deleting them).

[alucryd]

BTW, there's still an issue with completion calculation on arcade systems I need to address. But I already made it so parent files in split mode are marked as ignored now, they are missing only if in non-merged mode (if actually missing).

[alucryd]

And with https://github.com/alucryd/oxyromon/commit/c2756666fadacbadeec915e19a8d54b744e65896 arcade completion should finally be up to snuff. You'll probably want to use rebuild_roms back and forth between split and non-merged once to trigger the completion computation if you currently have completion errors in the web UI.

[TylerVigario]

Closing as the original issue has been resolved