search

unexpectedpanda / retool

Retool: a better filter tool for Redump and No-Intro DAT files.
BSD 3-Clause "New" or "Revised" License
368 stars 24 forks source link

System user filter exclude not working #238

Closed BobbyBoyGaming closed 1 year ago

BobbyBoyGaming commented 1 year ago

Dat

No-Intro's "Redump Custom": Nintendo - GameCube - NKit RVZ [zstd-19-128k] (2023-01-09)

Redump.org: Nintendo - GameCube (2023-03-05 21-29-39)

Issue

The USA title "Harry Potter and the Sorcerer's stone" is included and therefore The European variants "Harry potter and the philosopher's stone" should be excluded completely. As well as the Japanese version of the game "Harry Potter: To Kenja no Ishi" (or whatever, I don't speak Japanese)

image
unexpectedpanda commented 1 year ago

This works fine for me with both the original Redump GameCube DAT and the No-Intro custom DAT. The titles are already in the clone list.

Have you downloaded the clone lists? File > Update clone lists in the GUI, or retool.py --update from the command line.

If you have downloaded the clone lists, do you have an include user filter that might be getting in the way?

BobbyBoyGaming commented 1 year ago

I have a backup Log.txt file of my results from when I produced this dat. Nintendo - GameCube - NKit RVZ [zstd-19-128k] (2023-01-09) (Retool 2023-03-07 03-11-39) [-aABbcdekMmoPpruv] log.txt

No user filters, in fact, my user filters weren't even working at all when I did try to use them. I will try updating the clone lists again and see if this helps. Very strange, because I did already produce 1G1R DATs and they did work well for the most part.

At one point I had this list in the "Exclude" list but none of these key words (Partial matches) did anything, and I also don't see how any of these would have messed up the harry potter games. But I am pretty sure I produced these particular Nkit Gamecube retool dats without any of these filters anyway, because I hadnt started playing with the exclusion stuff until later.

Pokemon Barbie Mary-Kate Yu-Gi-Oh Naruto Digimon Dragon Ball Puzzle Pool Sudoku Disney's Disney Disney-Pixar Pixar Dreamworks Nickelodeon Ben 10 Dance Dancing Beat Deal or No Deal Who Wants To Be A Millionaire Singstar Jumpstart Kidz karaoke Sesame Street Pet Petz Harvest Moon Bratz Rock Band Guitar Hero Puyo Puff Girls Final Fantasy Online Bust-A-Move Rocksmith Persona Hello Kitty My

unexpectedpanda commented 1 year ago

No problems in the excludes as far as I can see. Let me know how you go with the clone lists. If you've got Nintendo - GameCube (Redump).json in the clonelists subfolder, then there's something else afoot.

BobbyBoyGaming commented 1 year ago

Yeah so the Harry Potter issue got fixed by updating the clones lists. I thought they had been downloaded after running the program for the first time. I will have to redo all my dats just in case.

But the User filter lists are still not working for me. This is the new Log.txt after updating the clone lists

Nintendo - GameCube - NKit RVZ [zstd-19-128k] (2023-01-09) (Retool 2023-03-09 01-39-17) [-AabBcDdekMmopPruv] log.txt

The harry potter part is fixed, but I still have all the Barbie games, hello kitty games and harvest moon games.

image

Here is a screenshot of the exclusions (full list is the previous post)

image

The output log.txt, does not appear to report the results of the user filters, I am not sure if this is because it has not been implemented into the log.txt production or because the program is not applying the filters as I would expect it to. Not sure what is going on.

image image

I put the new dat into romvault to actually inspect its contents and I see that the Harvest Moon, Hello Kitty, and all the Disney/Disney's/Disney-Pixar games are still there. None of the partial match filters worked for me.

unexpectedpanda commented 1 year ago

Ah I meant I couldn't see any issue with the excludes in relation to the Harry Potter problem. I didn't test the exclude functionality.

I've checked now and the global exclude user filter is working for me, but looks like the system include/exclude isn't working. I'll look into it.

unexpectedpanda commented 1 year ago

Edit: Looks like I didn't filter for the name change for dats.site and No-Intro Redump custom DATs in regards to system user filters.

System user filters work fine with the Redump DAT. I'll add an exception in.

unexpectedpanda commented 1 year ago

This should be fixed now in beta 2. Let me know how it goes.

BobbyBoyGaming commented 1 year ago

Cool, can't test it now but I will when I get home later

BobbyBoyGaming commented 1 year ago

There's still something sketchy about the filters, I dont know how I might be using them incorrectly but they simply do not delete the titles I want them to.

For example (same filter list as shown above, however I removed the last entry "My" it was not specific enough). I wanted "My" to remove "My horse" "My pet" and other dumb stuff like that, but it instead removed the "Myst" series, which would indicate that this filter actually works. However, when you look at every other user filter entry, none of them did anything.

I still have all the Yu-Gi-Oh! games, all the Final Fantasy games. For some reason the Hello Kitty filter did work, but I still have all the Disney games...

It seems like the partial match is only for a single word not "partial match for a whole title [group of words]"? I don't know whats going on. I would play around with the regular expression stuff but its such a pain, that I am trying to avoid it as much as possible. Because by the time I learn how to use regex properly, I would have just removed each title by hand myself.

Retool Exceptions Key Words.txt

Sony - PlayStation 2 (2023-03-07 01-04-26) (Retool 2023-03-10 12-57-45) [-aAbBcDdekmMoPpruv] log.txt

I attched my exclusion list text file as well as a PS2 Redump DAT I tried to apply those filters to (under the system-specific settings tab). I tried applying the same settings both as "global" and as "system specific" and I'm getting very similar issues with both when it comes to difficulty removing certain titles.

In my filter list you will see I use many variants of "Disney" so that it will take whatever it can and use it. I put in "Disney", "Disney's", "Disney-Pixar", "Pixar", "Dreamworks", and "DreamWorks" just in case that upper-case W throws the thing off.

but my log.txt still has all the disney games included.

unexpectedpanda commented 1 year ago

I don't think you're using them incorrectly. I'll dig into it.

unexpectedpanda commented 1 year ago

All the defined excludes should now be removed as intended in beta 3. Give it a try and let me know.

A few tips on partial matches:

BobbyBoyGaming commented 1 year ago

I tested the filters on Beta 4 and they do seem to work almost flawlessly now. I did notice a game called Tetris World was being removed for no apparent reason, but I deleted those logs and dats. I will keep playing around with it, and if I notice anything big I will post it here, however the filters do seem to work well now.

unexpectedpanda commented 1 year ago

With the exclude list you've provided, Tetris World survives for me. I'll close this up for now, but if you find another problem feel free to start a new issue.