No-Intro Nintendo DS DAT - Retool duped all listings (Update: No-Intro problem, default settings/daily pack includes two of everything)

unexpectedpanda / retool

Retool: a better filter tool for Redump and No-Intro DAT files.

BSD 3-Clause "New" or "Revised" License

343 stars 22 forks source link

No-Intro Nintendo DS DAT - Retool duped all listings (Update: No-Intro problem, default settings/daily pack includes two of everything) #293

Closed Nonstiq closed 7 months ago

Nonstiq commented 7 months ago

As Retool processed the latest available No-Intro Nintendo DS DAT, it duplicated all entries with a "(Dupe 1)" in the DAT entry. Only happened to this one DAT; the other No-Intro DATs I processed at the same time were trimmed as normal.

My ROM manager (ROMVault) then processed this DAT, and duplicated all of the files (all the dupes had "_0" as a suffix in the filename).

Is this some fault of the No-Intro DAT itself? Or with Retool's clone list?

unexpectedpanda commented 7 months ago

This should only happen if the source DAT file has duplicate filenames in it.

I can't replicate the issue here on Retool 2.01.7, but there are two ways that come to mind as to how this could occur:

The Retool code is doing something odd (clone lists shouldn't be responsible for this).
The source DAT file has a problem.

Let's start with step one. Attach the following files here so I can better understand your setup (assuming you're using the GUI):

The input DAT file
config\internal-config.json
config\user-config.yaml
If it exists, config\systems\Nintendo - Nintendo DS (No-Intro).yaml

Keiyuu commented 7 months ago

I'm having the exact same problem with the DS DAT. Nothing about my config has changed, only thing I did is updated clone lists and downloaded the new DS DAT. Version is 2.01.7. I attached my files. files.zip

unexpectedpanda commented 7 months ago

Oh yeah. It really hates that DAT, but not the one I downloaded with a custom configuration. Interesting. I'm going to have to prod a little.

unexpectedpanda commented 7 months ago

Looks like the DAT in the daily pack from Dat-o-matic is broken. Every title is duplicated.

The DAT is about 5.7MB that you supplied -- the actual size should be around 2.8MB. If you download the DAT individually, you'll get the non-broken version.

Nonstiq commented 7 months ago

If you download the DAT individually, you'll get the non-broken version.

I did download it individually... I'll try again. EDIT: Ahh, I downloaded it individually but with the default options. I did it again and changed one option, and it didn't have all the dupes.

Nonstiq commented 7 months ago

Even though this is an issue with the DAT, is there any way for Retool in the future to automatically detect and delete such dupes when it processes a DAT?

unexpectedpanda commented 7 months ago

Ah default settings too -- thanks for the extra info. I'll keep this issue open until No-Intro fixes it so people know it's not Retool's fault.

I can look at adding another level of dedupe that's for an entire game node in a DAT file and not just the title's name (which is the behavior you're seeing here with the rename to (Dupe #)). I'll need to find a place to put it where it's not that expensive on CPU or RAM.

Keiyuu commented 7 months ago

I can look at adding another level of dedupe that's for an entire game node in a DAT file and not just the title's name (which is the behavior you're seeing here with the rename to (Dupe #)). I'll need to find a place to put it where it's not that expensive on CPU or RAM.

Honestly, if this could potentially have a performance impact or affect PC resources I'm not sure if it's necessarily needed for the very rare occasion where a DAT has a temporary error. I'd personally rather just wait with updating until they fixed it.

unexpectedpanda commented 7 months ago

I can look at adding another level of dedupe that's for an entire game node in a DAT file and not just the title's name (which is the behavior you're seeing here with the rename to (Dupe #)). I'll need to find a place to put it where it's not that expensive on CPU or RAM.

Honestly, if this could potentially have a performance impact or affect PC resources I'm not sure if it's necessarily needed for the very rare occasion where a DAT has a temporary error. I'd personally rather just wait with updating until they fixed it.

No need to worry, you're not going to notice much of a difference unless there are actually dupes in the DAT (and even then, the impact will be small). This is just a typical decision making process whenever a software change is made. How and when you perform a task affects resource usage differently.

unexpectedpanda commented 7 months ago

Addressed in 2.01.8.