[FEATURE] Web UI filtering excluded ROMs from imported DATs

[aarongeorge]

I like to import the full No-Intro DATs without filtering (as there have been multiple occasions when the filtered DATs aren't categorised correctly).

I realised however that when I spin up the server and look at the web ui, there doesn't seem to be a filter that excludes in the same way that the DISCARD_FLAGS and DISCARD_RELEASES config options do when you sort ROMs. Instead they are showing up as Incomplete which makes finding actually incomplete/missing roms difficult.

I know it's not on your fault that the filtered DAT files can have issues, but if oxyromon can handle filtering in the Web UI based on the DISCARD_* config options, that would be amazing!

[alucryd]

I'm not sure I understand, if you use the DISCARD settings and have already sorted your roms, these should be marked as ignored (in gray) in the web UI, clicking the Ignored button up top should make them disappear.

Or are you saying you don't want to use the DISCARD settings, and would like a way to do it on the fly from the web UI without altering the database?

[aarongeorge]

Sorry I'll try to explain it a bit better

If I have a DAT that includes (Unl), (Homebrew) etc ROMs but my actual ROM set doesn't have them, even if I have these flags/releases marked as DISCARD, the Web UI shows them as "Incomplete" and clicking "Ignored" doesn't hide them

So with this config:

DISCARD_FLAGS = Aftermarket,Debug
DISCARD_RELEASES = Beta,Proto,Sample,Demo,Hack,Bootleg,Homebrew
HASH_ALGORITHM = CRC
REGIONS_ALL =
REGIONS_ALL_SUBFOLDERS = ALPHA
REGIONS_ONE = EU,US,AU,JP
REGIONS_ONE_SUBFOLDERS = ALPHA
ROM_DIRECTORY = XXXXXX
TMP_DIRECTORY = XXXXXX

And with "Ignored" filtered out, I'm still getting these ROMs showing up. The only way to hide them is to filter out "Incomplete" but that hides potentially missing ROMs

[alucryd]

Thanks for the clarification. It sounds like you haven't run sort-roms. When you import dats and roms, they are initially unsorted, and the various discard flags are not applied. That could change in the future, but for now the sorting is a separate step, and only sort-roms will actually apply any filtering. Please try it and let me know, here's what it should look like:

[aarongeorge]

Sorry for the delay

Everytime I run sort-roms I get the following error after selecting the system and typing Y to start the sort

Proceed? yes
thread 'main' panicked at 'Error while updating romfile with id 16974', src/database.rs:1450:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[alucryd]

Hmm, that's a first. I'll whip up some debug logging to know more. What's your OS and architecture? I'll prepare a test build.

[aarongeorge]

macOS Ventura 13.4 (22F66) Apple M1 Max

[alucryd]

Thanks, I'll see if I can cross-build that from linux, if not I'll just push to develop for you to build, will be nice to have some debug infrastructure anyway.

[aarongeorge]

I was using oxyromon 0.14.0-beta, just building develop now to see what's going on

[aarongeorge]

Now getting:

Error: SimpleError { err: "Failed to copy \"/Emulation/Nintendo - Game Boy/2097 ROM Pack II (USA) (Test Program).gb\" to \"/Emulation/Nintendo - Game Boy/1G1R/#/2097 ROM Pack II (USA) (Test Program).gb\", could not copy `/Emulation/Nintendo - Game Boy/2097 ROM Pack II (USA) (Test Program).gb` to `/Emulation/Nintendo - Game Boy/1G1R/#/2097 ROM Pack II (USA) (Test Program).gb`" }

Interesting it's trying to copy a ROM that already exists in the 1G1R directory, but not in the root directory. (Was copied previously when sort-roms was working)

[alucryd]

Copy is a fallback when moving doesn't work (for instance rust can't mv between different filesystems, it only works inside the same fs), that just means it failed to move the file.

The issue is caused by the previous error you had, it caused oxyromon to rollback the database transaction, but files were already moved.

Please run purge-roms -m, this will delete missing roms from the database, and then reimport your roms, should repair the database, and let you run sort-roms afterwards.

I need to come up with a better way to handle rollbacks when sorting fails, I'll probably just revert file moves as well.

Edit: BTW you can reimport the roms as is, no need to move them elsewhere in between.

[alucryd]

I'll be out for a couple hours, but I'll take a look at adding debug log afterwards if the develop branch HEAD still crashes on your machine.

[aarongeorge]

Appreciate the help mate. No rush. I'll try that now and get back to you

[aarongeorge]

That got me running again, thank you. Just have to remember to purge and sort before spinning up the server if I add or remove roms or change the config flags.

Thanks heaps

[alucryd]

Glad you're back on track! I don't think there was a code error, most likely the older SQLite driver in the 0.14 beta had some sort of bug.

FTR, you normally don't have to run purge-roms, -m is for these moments where the db might be out of sync with the fs, whether because of a bug, or files that were moved/deleted outside of oxyromon. -t is for physically deleting roms in the trash directories, and -o is for deleting files that no longer have any rom associated to them.

For the next release I'll also build Mac binaries, wasn't sure I had Mac users in my userbase. I believe M1 binaries have to be signed though (unlike x64), I don't think the rust toolchain does that by default, but rcodesign will come to the rescue.