Freezing while processing "Nintendo - Nintendo 3DS (Digital) (CDN) (20221017-043143).dat"

[coreyemtp]

Describe the bug While processing the "Nintendo - Nintendo 3DS (Digital) (CDN) (20221017-043143).dat" file, Retool locks up at the "* Finding titles in regions... Japan [Finishing up...]" phase/cli output.

To reproduce Choose the aforementioned DAT, process with standard settings with the following exceptions:

• Bad Dumps and Demos selected under exclusions
• Output lists of titles kept/removed and 1G1R name titles

Expected behavior Process to completion

Operating system

• [x] Windows 11
• [ ] Ubuntu
• [ ] Other (please specify)

Retool edition

• [x] retool-gui.exe
• [ ] retool.py
• [ ] retool-gui.py

Retool version 1.18

Are you using custom global or system filters? If so, list them below No

Global excludes:

Global includes:

System excludes:

System includes:

What other settings are you using?

• Bad Dumps and Demos selected under exclusions
• Output lists of titles kept/removed and 1G1R name titles

Additional context

[unexpectedpanda]

Oof, yep, this is clearly a problem. I suspect it's not a hang, but is so excruciatingly slow it might as well be. I gave up at the 30 minute mark for finishing up the Japan region.

You can tell there's a horrible Big-O complexity issue by the way the percentage counter slows down as it progresses when processing the Japanese region. Basically when I wrote this bit of Retool I didn't make the right design choices to handle DAT files of this size and complexity.

Unfortunately it's too big an architectural problem to justify fixing in v1. The 3DS CDN DAT will have to wait until v2 is ready 😔 No ETA on that, but the good news is that it currently finishes the 3DS CDN DAT in 150 seconds with your options. Still not great, but definitely better than the status quo.

[coreyemtp]

That DAT is a biggie, that's for sure. It was trivial to have my scripts move it out of the way before Retool works its magic, so absolutely not worth the distraction from v2 as it's the only one with the problem. And 150 seconds is just fine with me! I'm sure it's made up in spades from the savings on all the other DATs.

[muggsyd]

Yes, I should have logged a big about this. I have implemented additional steps in my scripts to remove the 3DS file as well. Can't wait for V2

[Ranmizuno]

It wasn't a freeze in my case. At first glance, it looked as if it had frozen, but I left it as it was and all the processing was completed in 3 days (72 hours). (If I left the ReTools main body and command prompt or terminal displayed, it ended successfully in 3 days. I don't know where ReTools uses machine power, but if it's CPU Only, if I raise the CPU name, it's i7 8770K.

[Ranmizuno]

Of course, the ReTools title bar would show "Not Responding", but every few hours the "Not Responding" text would disappear. It will be displayed again, but since "no response" is not displayed on the command prompt or terminal, I presume that the internal processing is progressing at a low pace, and I was prepared for about a week, but it ended in 3 days. That's the story.

[Ranmizuno]

I tried again and found that the text in the terminal would mention the time it took. The filter settings are only for these two places, and the others are left as default.

Since it came out with 72776.35 seconds, it seems to be 20 hours. The CPU I mentioned earlier is incorrect and the correct one is the i7 8700K.

[unexpectedpanda]

That's a lot of patience waiting for that to finish! :)

CPU power isn't really the issue here, it's probably just a series of nested loops in the software that are doing way too much work. Higher efficiency is one of the reasons I started on the v2 path.

By coincidence, Retool has been entirely written on an 8700K. So it looks like v2 finishes that DAT about 485x faster. It's still quite some way from being finished though.

[unexpectedpanda]

v2 is now out, and handles things just fine. It's slower than the internal version I was testing on, but it still finishes this DAT a lot quicker :)