search

unexpectedpanda / retool

Retool: a better filter tool for Redump and No-Intro DAT files.
BSD 3-Clause "New" or "Revised" License
349 stars 23 forks source link

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 15072: character maps to <undefined> #290

Closed CmdrShepardsPie closed 9 months ago

CmdrShepardsPie commented 9 months ago

Describe the bug Retool GUI console reports character encoding errors when reading DAT files (I think?) during a manually triggered "Update clone lists"

To reproduce I have a bunch of DATs from No-Intro, TOSEC, OpenGood, FinalBurn Neo, and Mame 0.140 and 0.260. I have them loaded into the tool and I click "Update clone lists" from the File dropdown. In the console it reports a bunch of errors reading the file or decoding it or something.

Expected behavior I don't know to be sure, but not errors!

Operating system

Retool edition

Retool version 2.01.5

Are you using custom global or system filters? If so, list them below

Global excludes: All but "Games" (Games unchecked, all other items unchecked)

Global includes: N/A?

System excludes: N/A

System includes: N/A

What other settings are you using? "Also output a DAT file of all removed titles" "Also output a list of what titles have been kept and removed"

Additional context Error log (this is a small sample, but happens on many more DATs from multiple sources (not just TOSEC))

Retool 2.01.5
-------------
* Downloading config/internal-config.json... done.
* Downloading datafile.dtd... done.
* Checking for updated clone lists...
* Checking for updated metadata files...

Retool has had an unexpected error. Please raise an issue at
https://github.com/unexpectedpanda/retool/issues, attaching
the DAT file that caused the problem and the following trace:

Traceback (most recent call last):
  File "retoolgui.py", line 175, in run
  File "retool.py", line 130, in main
  File "modules\clonelists.py", line 922, in update_clonelists_metadata
  File "modules\clonelists.py", line 885, in get_updates
  File "encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 15072: character maps to <undefined>

The error occurred on this file:
C:\Users\<xxx>\emulation\retool-2.01.5-win-x86-64\DATs\TOSEC\TOSEC-PIX\VTech V.Flash Home Edutainment System - Manuals - Games (TOSEC-v2012-07-21_CM).dat

Retool 2.01.5
-------------
* Downloading config/internal-config.json... done.
* Downloading datafile.dtd... done.
* Checking for updated clone lists...
* Checking for updated metadata files...

Retool has had an unexpected error. Please raise an issue at
https://github.com/unexpectedpanda/retool/issues, attaching
the DAT file that caused the problem and the following trace:

Traceback (most recent call last):
  File "retoolgui.py", line 175, in run
  File "retool.py", line 130, in main
  File "modules\clonelists.py", line 922, in update_clonelists_metadata
  File "modules\clonelists.py", line 885, in get_updates
  File "encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 15072: character maps to <undefined>

The error occurred on this file:
C:\Users\<xxx>\emulation\retool-2.01.5-win-x86-64\DATs\TOSEC\TOSEC-PIX\VTech V.Flash Home Edutainment System - Manuals - Hardware (TOSEC-v2012-07-21_CM).dat
unexpectedpanda commented 9 months ago

Retool only supports No-Intro and Redump DATs. No other release group/emulator DATs are supported. If you use unsupported DATs, you're likely to get a crash.

Now as to why an update operation is crashing which should be entirely separate from the DATs in question, that's something that I'll investigate when I've returned from break.

CmdrShepardsPie commented 9 months ago

Good to know, Thank you! Now that I'm using Retool correctly, it's helping a lot with managing my console ROMs, but I still need something similar for my arcade ROMs since it (apparently) isn't made for that. :)

unexpectedpanda commented 9 months ago

Arcade really is a different beast and requires a much more hands on approach. Here's a guide for trimming your MAME set that you might find useful.

unexpectedpanda commented 9 months ago

Turns out this was a metadata content problem, and not at all related to the DATs in the list. I'd recently added localized title names, and it was causing havoc. I'm surprised no one else reported it 😅

It'll be fixed in 2.01.7, just waiting on feedback on another bug.