search

watermelonwolverine / fvttoptimizer

Optimizes image files by converting them to webp while also updating all references.
MIT License
20 stars 1 forks source link

Program fails when a db file contains a byte UTF-8 can't decode. #2

Open YourDNDPleasesMe opened 2 years ago

YourDNDPleasesMe commented 2 years ago

I got the following message in powershell:

`PS C:\FVTTOptimizer> .\fvttoptimizer.exe --verbose-info --skip-webp --recursive "C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja" root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map' root - INFO - Optimizing C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. Decreased file size by 83% root - ERROR - Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

An internal error occurred: Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

Please file a bug report on https://github.com/watermelonwolverine/fvttoptimizer/issues`

YourDNDPleasesMe commented 2 years ago

image In case it helps, here's the folder where it created one .webp before failing.

watermelonwolverine commented 2 years ago

Interesting. It's not failing on the conversion part but on the part where it updates the databases. It can't read the file with UTF-8. Are you using special characters like Ð?

YourDNDPleasesMe commented 2 years ago

Not that I am aware of! Certainly not intentionally. If it processes files alphabetically I would assume the next file would be "Banner 8.png", which in the screenshot doesn't look lik eit has any special characters.

watermelonwolverine commented 2 years ago

It's not necessarily a problem with the file names, could be anything within foundry. Did you use any other tools to manipulate .db files? Also did you shut down foundry before optimizing?

After trying out a bit there are two possibilities:

  1. I'm completely wrong in my assumption that FoundryVTT encodes its db files in UTF-8
  2. One of your db file got corrupted. It fails at the first character, so that one isn't out of the question as the first character should be always {

If possible could you try and find out which db file is causing this issue and attach it?

To find out which db file is causing this issue use the --verbose-debug option. It will tell you at which db it fails, should be the last db it's trying to update before the exception.

YourDNDPleasesMe commented 2 years ago

I'm not aware of any tool that would have edited databases, nor am I aware of how to do so. Probably safe there.

I did shut down Foundry first.

I will check this tonight with the --verbose-debug flag. Thanks for looking at this!

YourDNDPleasesMe commented 2 years ago

I shut down Foundry, ran the following:

PS C:\FVTTOptimizer> .\fvttoptimizer.exe --verbose-debug --skip-webp --recursive "C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja"

root - DEBUG - Got arguments ['C:\FVTTOptimizer\fvttoptimizer.exe', '--verbose-debug', '--skip-webp', '--recursive', 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja'] root - DEBUG - Reading config from: C:\FVTTOptimizer\fvttoptimizer.conf root - DEBUG - Running with target_path='C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja', config='{'_RunConfigprogram_config': {'_ProgramConfigImplabs_path_to_foundry_data': 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data'}, 'skip_existing': False, 'skip_webp': True, 'recursive': True, 'quality': 75, 'override_percent': 25}' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map' root - ERROR - Cannot convert C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.webp already exists. Cannot convert C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.webp already exists.

YourDNDPleasesMe commented 2 years ago

The above didn't seem to be the same error, so I deleted the .webp it was complaining about, and ran it again:

PS C:\FVTTOptimizer> .\fvttoptimizer.exe --verbose-debug --skip-webp --recursive "C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja" root - DEBUG - Got arguments ['C:\FVTTOptimizer\fvttoptimizer.exe', '--verbose-debug', '--skip-webp', '--recursive', 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja'] root - DEBUG - Reading config from: C:\FVTTOptimizer\fvttoptimizer.conf root - DEBUG - Running with target_path='C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\Worlds\bakja', config='{'_RunConfigprogram_config': {'_ProgramConfigImplabs_path_to_foundry_data': 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data'}, 'skip_existing': False, 'skip_webp': True, 'recursive': True, 'quality': 75, 'override_percent': 25}' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets' root - INFO - Optimizing directory 'C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map' PIL.PngImagePlugin - DEBUG - STREAM b'IHDR' 16 13 PIL.PngImagePlugin - DEBUG - STREAM b'pHYs' 41 9 PIL.PngImagePlugin - DEBUG - STREAM b'iTXt' 62 2806 PIL.PngImagePlugin - DEBUG - STREAM b'IDAT' 2880 13529 PIL.Image - DEBUG - Importing BlpImagePlugin PIL.Image - DEBUG - Importing BmpImagePlugin PIL.Image - DEBUG - Importing BufrStubImagePlugin PIL.Image - DEBUG - Importing CurImagePlugin PIL.Image - DEBUG - Importing DcxImagePlugin PIL.Image - DEBUG - Importing DdsImagePlugin PIL.Image - DEBUG - Importing EpsImagePlugin PIL.Image - DEBUG - Importing FitsImagePlugin PIL.Image - DEBUG - Importing FitsStubImagePlugin PIL.Image - DEBUG - Importing FliImagePlugin PIL.Image - DEBUG - Importing FpxImagePlugin PIL.Image - DEBUG - Image: failed to import FpxImagePlugin: No module named 'olefile' PIL.Image - DEBUG - Importing FtexImagePlugin PIL.Image - DEBUG - Importing GbrImagePlugin PIL.Image - DEBUG - Importing GifImagePlugin PIL.Image - DEBUG - Importing GribStubImagePlugin PIL.Image - DEBUG - Importing Hdf5StubImagePlugin PIL.Image - DEBUG - Importing IcnsImagePlugin PIL.Image - DEBUG - Importing IcoImagePlugin PIL.Image - DEBUG - Importing ImImagePlugin PIL.Image - DEBUG - Importing ImtImagePlugin PIL.Image - DEBUG - Importing IptcImagePlugin PIL.Image - DEBUG - Importing JpegImagePlugin PIL.Image - DEBUG - Importing Jpeg2KImagePlugin PIL.Image - DEBUG - Importing McIdasImagePlugin PIL.Image - DEBUG - Importing MicImagePlugin PIL.Image - DEBUG - Image: failed to import MicImagePlugin: No module named 'olefile' PIL.Image - DEBUG - Importing MpegImagePlugin PIL.Image - DEBUG - Importing MpoImagePlugin PIL.Image - DEBUG - Importing MspImagePlugin PIL.Image - DEBUG - Importing PalmImagePlugin PIL.Image - DEBUG - Importing PcdImagePlugin PIL.Image - DEBUG - Importing PcxImagePlugin PIL.Image - DEBUG - Importing PdfImagePlugin PIL.Image - DEBUG - Importing PixarImagePlugin PIL.Image - DEBUG - Importing PngImagePlugin PIL.Image - DEBUG - Importing PpmImagePlugin PIL.Image - DEBUG - Importing PsdImagePlugin PIL.Image - DEBUG - Importing SgiImagePlugin PIL.Image - DEBUG - Importing SpiderImagePlugin PIL.Image - DEBUG - Importing SunImagePlugin PIL.Image - DEBUG - Importing TgaImagePlugin PIL.Image - DEBUG - Importing TiffImagePlugin PIL.Image - DEBUG - Importing WebPImagePlugin PIL.Image - DEBUG - Importing WmfImagePlugin PIL.Image - DEBUG - Importing XbmImagePlugin PIL.Image - DEBUG - Importing XpmImagePlugin PIL.Image - DEBUG - Importing XVThumbImagePlugin root - INFO - Optimizing C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Animal Market Stall.png. Decreased file size by 83% root - DEBUG - Replacing references to worlds/bakja/Assets/Map/Animal%20Market%20Stall.png with worlds/bakja/Assets/Map/Animal%20Market%20Stall.webp in C:\Users\Scott-Server\AppData\Local\FoundryVTT\Data\worlds\bakja\Assets\Map\Thumbs.db root - ERROR - Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

An internal error occurred: Traceback (most recent call last): File "cli_wrapper\main.py", line 235, in main File "cli_wrapper\main.py", line 229, in do_run File "cli_wrapper\main.py", line 81, in perform_optimization_with File "fvttoptimizer\optimizer.py", line 42, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 38, in optimize File "fvttoptimizer__directory_optimizer.py", line 35, in optimize File "fvttoptimizer__file_optimizer.py", line 34, in maybe_optimize File "fvttoptimizer__file_optimizer.py", line 48, in maybe_optimize2 File "fvttoptimizer__file_optimizer.py", line 68, in optimize File "fvttoptimizer__file_optimizer.py", line 100, in __replace_file_with_webp File "fvttmv\update\references_updater.py", line 22, in replace_references File "fvttmv\update\references_updater.py", line 34, in _do_replace_references File "fvttmv\update__references_updater_file.py", line 21, in replace_references_in_file File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

Please file a bug report on https://github.com/watermelonwolverine/fvttoptimizer/issues

YourDNDPleasesMe commented 2 years ago

I assume this is the correct file you want, let me know if I'm wrong (renamed thumbs.db to thumbs.txt so I could upload it): Thumbs.txt

watermelonwolverine commented 2 years ago

That seems to be a very old thumbnail cache from an Windows XP. Does that ring a bell?

YourDNDPleasesMe commented 2 years ago

Not at all. This has always been a Win10 machine. The only cached images in it are from the past 2 years, running FoundryVTT.