search

Wolverator / CatSystem2-Simple-Translating-Tools

Single-click tools to "extract movies, images and text right to editable state" and "pack all it back". Or as simple as I am able to make it :D (WIP)
16 stars 1 forks source link

UnicodeDecodeError: 'shift_jis' codec can't decode byte 0xef in position 261: illegal multibyte sequence #2

Closed pputnik closed 1 year ago

pputnik commented 1 year ago

Hello, "1) unpack.bat" generates the following error:

...
After copying all files into this folder press Enter...

Copying files into 'extracted' folder and unpacking 'int'-archives may take up to 1 minute per file...
Copying scene.int...
Copying config.int...
Copying update00.int...
Copying cs2.exe...
Processing scene.int...
scene.int: using v_code "AMN-018INN67"
Processing config.int...
config.int: using v_code "AMN-018INN67"
Processing update00.int...
update00.int: using v_code "AMN-018INN67"
Sorting extracted files...
ERROR - Traceback (most recent call last):
  File "C:\Program Files (x86)\Steam\steamapps\common\NEKOPARA Vol. 3\tools\unpack.py", line 439, in <module>
    sort_resulting_files()
  File "C:\Program Files (x86)\Steam\steamapps\common\NEKOPARA Vol. 3\tools\unpack.py", line 378, in sort_resulting_files
    df0 = pandas.read_csv(nametable_csv, encoding='ShiftJIS')
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 912, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 577, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 1407, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\io\parsers\readers.py", line 1679, in _make_engine
    return mapping[engine](f, **self.options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 93, in __init__Steam\steamapps\common\NEKOPARA Vol. 3>"1) unpack.bat"
    self._reader = parsers.TextReader(src, **kwds)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas\_libs\parsers.pyx", line 548, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas\_libs\parsers.pyx", line 637, in pandas._libs.parsers.TextReader._get_header
  File "pandas\_libs\parsers.pyx", line 848, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas\_libs\parsers.pyx", line 859, in pandas._libs.parsers.TextReader._check_tokenize_status
  File "pandas\_libs\parsers.pyx", line 2017, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'shift_jis' codec can't decode byte 0xef in position 261: illegal multibyte sequence

Removing temporary files...
Done! Program will be closed now.
Press any key to continue . . .

Instead of ShiftJIS I tried (according to doc https://docs.python.org/3/library/codecs.html#standard-encodings):

NekoPara vol.3 from Steam, ver 1.0 cs2.exe ver 2.6.1.76

Could you help, please?

pputnik commented 1 year ago

p.s. guessed out that utf_8 works here. Maybe, it makes sense to add some try...Exception?

Wolverator commented 1 year ago

Yes, a few hrs ago i've discovered a some my mistakes in logic i've put into reading .csv files, and your issue tells me about even one more :D

I will see what i can do. Ideally i could try to determine .csv encoding and use it.

All my problems with it is that i've coded this tool just to translate Grisaia, so there are high chances there will be some unexpected issues with other games.

But i'll try to address and test it on other games too.

Wolverator commented 1 year ago

Result was making code to try both of ShiftHIS and UTF-8 encodings while trying to read nametable.csv and selecting encoding that works.

Tested, worked on Nekopara3, Grisaia 1, 2 and 3.

Huge thank you for writing about found error! @pputnik