search

unexpectedpanda / retool

Retool: a better filter tool for Redump and No-Intro DAT files.
BSD 3-Clause "New" or "Revised" License
368 stars 24 forks source link

Issues filtering Nintendo - Nintendo 3DS (Encrypted) #280

Closed muggsyd closed 1 year ago

muggsyd commented 1 year ago

Describe the bug When using the following switches on the Nintendo - Nintendo 3DS (Encrypted) DAT, I go from over 2000 entries, to just 25. This used to work. It's been broken for a few releases and I am unsure the last version it worked on. This does not appear to affect the other No-Intro DATs I am running retool against.

To reproduce run retool.py with following switches -l --exclude aAbBcdDemMopPrv

Expected behavior I would expect filtering based on above switches

Operating system

Retool edition

Retool version 2.0.1.4

Are you using custom global or system filters? If so, list them below

Global excludes:

Global includes:

System excludes:

System includes:

What other settings are you using?

user-config.yaml includes. All other region/language are commented out

==============

LANGUAGE ORDER

==============

============

REGION ORDER

============

unexpectedpanda commented 1 year ago

With Nintendo - Nintendo 3DS (Encrypted) (20230406-211023).dat, I get 554 titles with your settings, in both Retool GUI and CLI on Windows. The same is true whether I use global or system settings for the filtering.

A quick spin on Ubuntu via WSL using CLI and global settings gives the same result.

user-config.yaml includes.

What includes?

I think the easiest way to diagnose this is to provide the following files:

muggsyd commented 1 year ago

Here are the files. When I remove the "-l" is seems to filter better. I didn't know where the logs files were kept (I'm running in docker) so I have copied the output into txt files. I have also include the json and yaml file you requested

retool.zip

unexpectedpanda commented 1 year ago

I wrote up a whole piece of the diagnostic I did thinking that I'd hit a dead end, but I see you've provided the answer, possibly by accident.

In your file bundle is your 3DS metadata file, which I didn't ask for, but is funnily enough the problem.

It's old and in the wrong format. You need to update your clone lists, which also grabs new metadata (--update from the command line).

Here's what that file should look like, with values as arrays instead of strings:

https://github.com/unexpectedpanda/retool-clonelists-metadata/blob/main/metadata/Nintendo%20-%20Nintendo%203DS%20(No-Intro).json

Edit: This has been an issue before, by the way. If the update doesn't get you the right version of the file, it's entirely possible your internal-config.json is also old and pointing to the wrong place.

muggsyd commented 1 year ago

That's strange. The 1st thing my script does it run a retool.py --update All those meta files were updated the last time I ran the script. I renamed the old meta file and re-ran and it appeared to work the 1st time. When I tried it again it once again failed. However the new and the old metafile appear to be identical [image: image.png] I also wiped out all my clonelist and metadata, so it all redownloaded

retool | retool | Done. Downloaded 346 files. retool | retool | retool | Retool 2.01.4 retool | ------------- retool | Processing folder... retool | Reading DAT file: "/retool/dats/No-Intro/Nintendo - Nintendo 3DS (Encrypted) retool | (20230724-091936).dat" retool | * Validating DAT file... done. retool | retool | | DAT DETAILS retool | | Description: Nintendo - Nintendo 3DS (Encrypted) retool | | Author/s: aci68, ajax16384, ajshell1, Arctic Circle System, Aringon, b2071988, Bent, retool | buckwheat, C. V. Reynolds, Connie, coraz, Datman, DeadSkullzJr, Densetsu, einstein95, Fonix, retool | Gefflon, Hiccup, InternalLoss, Jack, jimmsu, Just001Kim, Larsenv, MeguCocoa, MisterSheeple, retool | Money_114, NESBrew12, NovaAurora, PPLToast, psykopat, relax, Rifu, scorp256, SonGoku, retool | Tauwasser, TBemme, togemet2, u53r123, Xenirina, xprism, xuom2, ZeroSkill, zg, Zora retool | | URL: https://www.no-intro.org retool | | Version: 20230724-091936 retool | | Numbered dat: No retool |

I'm rebuilding my environment. It's possible an issue with my data as you pointed out

On Fri, 28 Jul 2023 at 08:34, unexpectedpanda @.***> wrote:

I wrote up a whole piece of the diagnostic I did and thinking I'd hit a dead end, but I see you've provided the answer, possibly by accident.

In your file bundle is your 3DS metadata file, which I didn't ask for, but is funnily enough the problem.

It's old and in the wrong format. You need to update your clone lists, which also grabs new metadata (--update from the command line).

Here's what that file should look like:

https://github.com/unexpectedpanda/retool-clonelists-metadata/blob/main/metadata/Nintendo%20-%20Nintendo%203DS%20(No-Intro).json

— Reply to this email directly, view it on GitHub https://github.com/unexpectedpanda/retool/issues/280#issuecomment-1654669530, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ2XWFO7VKGAOVALJLNDSVDXSLUGBANCNFSM6AAAAAA2ZQ4JGI . You are receiving this because you authored the thread.Message ID: @.***>

muggsyd commented 1 year ago

It actually looks like wiping my internal-config.json and my user-config.yaml and having them re-downloaded fixed it for me. This also replicated your results

retool | Stats: retool | o Original title count: 2,092 retool | - Clones removed: 559 retool | - Add-on titles removed: 0 retool | - Applications removed: 4 retool | - Audio titles removed: 0 retool | - Bad dumps removed: 0 retool | - BIOSes and other chips removed: 2 retool | - Bonus discs removed: 0 retool | - Coverdiscs removed: 0 retool | - Demos removed: 68 retool | - Educational titles removed: 0 retool | - Manuals removed: 0 retool | - Multimedia titles removed: 0 retool | - Pirate titles removed: 0 retool | - Preproduction titles removed: 9 retool | - Promotional titles removed: 0 retool | - Video titles removed: 0 retool | - Titles removed by language filters: 895 retool | retool | - Total titles removed: 1,537 retool | ------------------------------- retool | = Final title count: 555

The only other change once they had been downloaded was uncommenting English is user-config.yaml

Edit: Sorry I must have missed your post about the user-config and internal-config. This is possibly carried over from beta testing

unexpectedpanda commented 1 year ago

No worries :) Yep that definitely looks like internal-config.json was pointing to the wrong place.

I think I might be able to work around that for future users... that's at least two people that grabbed the first beta who have had the issue.

Out of interest, how are you dealing with storage in your container? I'm assuming the output file ends up in a folder shared with the host system?

Sounds like you're generating a user-config.yaml in the container itself.

I started messing with Docker based on your Dockerfile, but I imagine it'll be some time before I can wrap my head around how I should set everything up in terms of config files and output. I also didn't quite understand creating the retool user, but then it didn't seem to run as that user? Or is that what setting arg UID does? (I should put this in the Docker issue, but while I'm here...)

muggsyd commented 1 year ago

Yes, I am seeing actual clone list updates (I assumed they were pretty static) etc now that those config files were cleaned up

Here is my docker-compose.yml file for my volumes which are all local storage on my docker (except the last volume)

I see the log file is placed in the DAT output location. Anyway to have a switch to output it to a log folder? like /retool/logs/

On Fri, 28 Jul 2023 at 13:18, unexpectedpanda @.***> wrote:

No worries :) Yep that definitely looks like internal-config.json was pointing to the wrong place.

I think I might be able to work around that for future users... that's at least two people that grabbed the first beta who have had the issue.

Out of interest, how are you dealing with storage in your container? I'm assuming the output file ends up in a folder shared with the host system?

— Reply to this email directly, view it on GitHub https://github.com/unexpectedpanda/retool/issues/280#issuecomment-1654918134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ2XWFPBPYM25UMCL34U34DXSMVOZANCNFSM6AAAAAA2ZQ4JGI . You are receiving this because you authored the thread.Message ID: @.***>

unexpectedpanda commented 1 year ago

Cheers! Eventually I'll get something semi-working on the Docker front.

You can't currently route log output elsewhere, but I'll add it as a feature request.