search

wingtk / gvsbuild

GTK stack for Windows
GNU General Public License v2.0
459 stars 172 forks source link

UnicodeDecodeError trying parse vcvars bat-file. #368

Closed mhertz closed 4 years ago

mhertz commented 4 years ago

I hope it's okay I forward a report by other user here. I haven't the issue personally, but am then guessing it's a localization issue possibly? Affected user is using msvc 2019 buildtools just installed yesterday or so, so newest version, and py 3.8.3 and newly updated msys2, on win10 x64. All components installed and setup from unattended script I made, which affected user also installed from, so we have exact same setup, except maybe different locales where I have US.

This is exact error:

Checking Msvc tool
Running script "c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
Debug: vs env:ALLUSERSPROFILE -> [C:\ProgramData]
Traceback (most recent call last):
  File "build.py", line 48, in <module>
    args.func(args)
  File "C:\gtk-build\github\gvsbuild\gvsbuild\utils\parser.py", line 160, in do_build
    builder = Builder(opts)
  File "C:\gtk-build\github\gvsbuild\gvsbuild\utils\builder.py", line 86, in __init__
    self.__check_vs(opts)
  File "C:\gtk-build\github\gvsbuild\gvsbuild\utils\builder.py", line 369, in __check_vs
    l = l.decode('utf-8') if isinstance(l, bytes) else l
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 18: invalid start byte

I believe there's options for selecting to only install non-localised versions of msvc, from cmd-line options, so am thinking if that would help maybe, but still is then an issue which many would be bitten from seemingly.

I couldn't find previous reports for this here or online(vcvars parsing UnicodeDecodeError specifically), but as general issue there is several, and states to change encoding to e.g. windows-1252 or latin etc. It's seems 0x92 is a smart quote character from windows-1252, which some say can be problematic reading/decoding through utf-8, and some stated fixed by changing encoding to windows-1252, or resaving file in question in utf-8, but dunno, have no experience with such personally, but sounds like the issue however. Just strange something like that could occure i.e. other localization introducing smart/curly quotes???

Many thanks in advance for any feedback/help.

Ohh, this is full log of command-line run and output:

C:\gtk-build\github\gvsbuild>python build.py -d build --gtk3-ver=3.24 --archives-download-dir=C:\gtk-cache --vs-ver=16 --platform=x64 --vs-install-path="c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools" --python-dir=C:\python -k --enable-gi --py-wheel --py-egg --python-ver=3.8.3 enchant gtk gtk3-full pycairo pygobject lz4 --skip gtksourceview3,emeus,clutter,adwaita-icon-theme
Log directory C:\gtk-build\logs created
Debug: Version from file name:                 <- rustup-init.exe
Debug: Version from file name:3.7.2            <- cmake-3.7.2-win64-x64.zip
Debug: Version from file name:0.52.0           <- meson-0.52.0.zip
Debug: Version from file name:2.13.03          <- nasm-2.13.03-win64.zip
Debug: Version from file name:1.8.2            <- ninja-win-1.8.2.zip
Debug: Version from file name:5.4.0            <- nuget-5.4.0.exe
Debug: Version from file name:5.20.0           <- perl-5.20.0-x64.tar.xz
Debug: Version from file name:1.3.0            <- yasm-1.3.0-win64.exe
Debug: Version from file name:1.10             <- go1.10.windows-amd64.zip
Debug: Version from file name:3.36.0           <- adwaita-icon-theme-3.36.0.tar.xz
Debug: Version from file name:2.36.0           <- atk-2.36.0.tar.xz
Debug: Version from file name:1.16.0           <- cairo-1.16.0.tar.xz
Debug: Version from file name:1.26.4           <- clutter-1.26.4.tar.xz
Debug: Version from file name:1.22.6           <- cogl-1.22.6.tar.xz
Debug: Version from file name:2.1.28           <- cyrus-sasl-2.1.28.tar.xz
Debug: Version from file name:0.1.8            <- dcv-color-primitives-0.1.8.tar.gz
Debug: Version from file name:1.6.1            <- enchant-1.6.1.tar.xz
Debug: Version from file name:4.2.2            <- ffmpeg-4.2.2.tar.xz
Debug: Version from file name:2.13.0           <- fontconfig-2.13.0.tar.gz
Debug: Version from file name:098dd70cb1845b8c325ef4801c5f2e09e476b1ed <- freetype2-098dd70cb1845b8c325ef4801c5f2e09e476b1ed.tar.gz
Debug: Version from file name:2.40.0           <- gdk-pixbuf-2.40.0.tar.xz
Debug: Version from file name:0.19.7           <- gettext-0.19.7.tar.gz
Debug: Version from file name:2.64.2           <- glib-2.64.2.tar.xz
Debug: Version from file name:2.64.1           <- glib-networking-2.64.1.tar.xz
Debug: Version from file name:2.50.8           <- glib-openssl-2.50.8.tar.xz
Debug: Version from file name:1.64.1           <- gobject-introspection-1.64.1.tar.xz
Debug: Version from file name:3.36.0           <- gsettings-desktop-schemas-3.36.0.tar.xz
Debug: Version from file name:1.16.2           <- gstreamer-1.16.2.tar.xz
Debug: Version from file name:1.16.2           <- gst-plugins-base-1.16.2.tar.xz
Debug: Version from file name:1.16.2           <- gst-plugins-good-1.16.2.tar.xz
Debug: Version from file name:1.16.2           <- gst-plugins-bad-1.16.2.tar.xz
Debug: Version from file name:1.16.2           <- gst-python-1.16.2.tar.xz
Debug: Version from file name:2.24.31          <- gtk+-2.24.31.tar.xz
Debug: Version from file name:3.22.2           <- gtksourceview-3.22.2.tar.xz
Debug: Version from file name:2.6.4            <- harfbuzz-2.6.4.tar.xz
Debug: Version from file name:0.17             <- hicolor-icon-theme-0.17.tar.xz
Debug: Version from file name:2.0.14           <- jasper-2.0.14.tar.gz
Debug: Version from file name:0.12.1-20160607  <- json-c-0.12.1-20160607.tar.gz
Debug: Version from file name:1.4.4            <- json-glib-1.4.4.tar.xz
Debug: Version from file name:1.20             <- leveldb-1.20.tar.gz
Debug: Version from file name:3.4.1            <- libarchive-3.4.1.tar.xz
Debug: Version from file name:0.6.13           <- libcroco-0.6.13.tar.xz
Debug: Version from file name:1.5.3            <- libepoxy-1.5.3.tar.xz
Debug: Version from file name:0.3.1            <- libgxps-0.3.1.tar.xz
Debug: Version from file name:2.0.4            <- libjpeg-turbo-2.0.4.tar.gz
Debug: Version from file name:0.9.54           <- libmicrohttpd-0.9.54.tar.gz
Debug: Version from file name:1.6.37           <- libpng-1.6.37.tar.xz
Debug: Version from file name:2.40.20          <- librsvg-2.40.20.tar.xz
Debug: Version from file name:7.54.0           <- curl-7.54.0.tar.gz
Debug: Version from file name:2.70.0           <- libsoup-2.70.0.tar.xz
Debug: Version from file name:0.9.3            <- libssh-0.9.3.tar.xz
Debug: Version from file name:1.8.0            <- libssh2-1.8.0.tar.gz
Debug: Version from file name:4.1.0            <- tiff-4.1.0.tar.gz
Debug: Version from file name:1.35.0           <- v1.35.0.tar.gz
Debug: Version from file name:2.9.10           <- libxml2-2.9.10.tar.gz
Debug: Version from file name:1.2.0            <- libzip-1.2.0.tar.gz
Debug: Version from file name:2.1.0            <- LuaJIT-2.1.0-beta3.tar.gz
Debug: Version from file name:1.9.2            <- lz4-1.9.2.tar.gz
Debug: Version from file name:1.16.1           <- krb5-1.16.1-final.tar.gz
Debug: Version from file name:1.1.1g           <- openssl-1.1.1g.tar.gz
Debug: Version from file name:1.3.1            <- opus-1.3.1.tar.gz
Debug: Version from file name:0.4.31           <- orc-0.4.31.tar.xz
Debug: Version from file name:1.44.7           <- pango-1.44.7.tar.xz
Debug: Version from file name:0.38.4           <- pixman-0.38.4.tar.gz
Debug: Version from file name:190600_20161030  <- pa_stable_v190600_20161030.tgz
Debug: Version from file name:3.9.2            <- protobuf-cpp-3.9.2.tar.gz
Debug: Version from file name:1.3.2            <- protobuf-c-1.3.2.tar.gz
Debug: Version from file name:1.19.1           <- pycairo-1.19.1.tar.gz
Debug: Version from file name:3.36.0           <- pygobject-3.36.0.tar.xz
Debug: Version from file name:3310100          <- sqlite-autoconf-3310100.tar.gz
Debug: Version from file name:0.0.8            <- win-iconv-0.0.8.tar.gz
Debug: Version from file name:3.4              <- wing-v0.3.4.tar.gz
Debug: Version from file name:1.2.11           <- zlib-1.2.11.tar.xz
Debug: Options are:
  '_load_python': False,
  '_vs_path_auto': False,
  'archives_download_dir': 'C:\gtk-cache',
  'build_dir': 'C:\gtk-build',
  'capture_out': False,
  'cargo_opts': '',
  'check_hash': False,
  'clean': False,
  'clean_built': False,
  'configuration': 'release',
  'debug': True,
  'enable_gi': True,
  'fast_build': False,
  'ffmpeg_enable_gpl': False,
  'from_scratch': False,
  'git_expand_dir': 'C:\gtk-cache\git-exp',
  'gtk3_ver': '3.24',
  'keep': True,
  'keep_tools': False,
  'log_single': False,
  'log_size': 0,
  'make_zip': False,
  'msbuild_opts': '',
  'msys_dir': 'C:\Msys64',
  'ninja_opts': '',
  'no_deps': False,
  'patches_root_dir': 'C:\gtk-build\github\gvsbuild\patches',
  'platform': 'x64',
  'print_out': False,
  'projects': ['enchant', 'gtk', 'gtk3-full', 'pycairo', 'pygobject', 'lz4'],
  'py_egg': True,
  'py_wheel': True,
  'python_dir': 'C:\python',
  'python_ver': '3.8.3',
  'same_python': False,
  'skip': 'gtksourceview3,emeus,clutter,adwaita-icon-theme',
  'tools_root_dir': 'C:\gtk-build\tools',
  'use_env': False,
  'verbose': False,
  'vs_install_path': 'c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools',
  'vs_ver': '16',
  'win_sdk_ver': None,
  'zip_continue': False,
Cleaning up the build environment
Debug: windir -> c:\\windows
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\vc\tools\msvc\14.26.28801\bin\hostx64\x64'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide\commonextensions\microsoft\testwindow'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\msbuild\current\bin\roslyn'
Debug:    Skip 'c:\program files (x86)\windows kits\10\bin\10.0.18362.0\x64'
Debug:    Skip 'c:\program files (x86)\windows kits\10\bin\x64'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\msbuild\current\bin'
Debug: Add 'c:\windows\microsoft.net\framework64\v4.0.30319'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\tools'
Debug:    Skip 'c:\python'
Debug:    Skip 'c:\python\scripts'
Debug:    Skip 'c:\gtk-build\gtk\x64\release\bin'
Debug:    Skip 'c:\msys64\usr\bin'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\vc\tools\msvc\14.26.28801\bin\hostx64\x64'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide\commonextensions\microsoft\testwindow'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\msbuild\current\bin\roslyn'
Debug:    Skip 'c:\program files (x86)\windows kits\10\bin\10.0.18362.0\x64'
Debug:    Skip 'c:\program files (x86)\windows kits\10\bin\x64'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\msbuild\current\bin'
Debug:    Already present: 'c:\windows\microsoft.net\framework64\v4.0.30319'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\tools'
Debug:    Skip 'c:\python'
Debug:    Skip 'c:\python\scripts'
Debug:    Skip 'c:\gtk-build\gtk\x64\release\bin'
Debug:    Skip 'c:\msys64\usr\bin'
Debug:    Skip 'c:\msys64\usr\bin'
Debug:    Skip 'c:\msys64\usr\bin'
Debug: Add 'c:\windows\system32'
Debug: Add 'c:\windows'
Debug: Add 'c:\windows\system32\wbem'
Debug: Add 'c:\windows\system32\windowspowershell\v1.0'
Debug: Add 'c:\windows\system32\openssh'
Debug:    Skip 'c:\users\pétersasi\appdata\local\microsoft\windowsapps'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide\commonextensions\microsoft\cmake\cmake\bin'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide\commonextensions\microsoft\cmake\ninja'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide\commonextensions\microsoft\cmake\cmake\bin'
Debug:    Skip 'c:\program files (x86)\microsoft visual studio\2019\buildtools\common7\ide\commonextensions\microsoft\cmake\ninja'
Debug: Final path:
Debug:     c:\windows\microsoft.net\framework64\v4.0.30319
Debug:     c:\windows\system32
Debug:     c:\windows
Debug:     c:\windows\system32\wbem
Debug:     c:\windows\system32\windowspowershell\v1.0
Debug:     c:\windows\system32\openssh
Checking msys tool
Debug: patch: C:\Msys64\usr\bin\patch.exe
Checking Msvc tool
Running script "c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
Debug: vs env:ALLUSERSPROFILE -> [C:\ProgramData]
Traceback (most recent call last):
  File "build.py", line 48, in <module>
    args.func(args)
  File "C:\gtk-build\github\gvsbuild\gvsbuild\utils\parser.py", line 160, in do_build
    builder = Builder(opts)
  File "C:\gtk-build\github\gvsbuild\gvsbuild\utils\builder.py", line 86, in __init__
    self.__check_vs(opts)
  File "C:\gtk-build\github\gvsbuild\gvsbuild\utils\builder.py", line 369, in __check_vs
    l = l.decode('utf-8') if isinstance(l, bytes) else l
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 18: invalid start byte

Edit: When only searching unicodedecodeerror solely, here, then I found this earlier ticket, not related to this issue here specifically(vcvars), but does however suggest localisation issues could be occuring since happened before, though in other component(meson), and other unicode char issue:

https://github.com/wingtk/gvsbuild/issues/247

guruDanny67 commented 4 years ago

You probably had some windows environment with char > 128 (e.g. some accented, probably é or something similar like ò, à, ù ...) and this gives the decoding error.

We can ignore the wrong character but if the character goes in the vs installation path (this can happen if you install it in your document & setting folder and your user name is not all ascii char) we cannot do anything: you must reinstall it in the default path or in a 'old' one (the old ones are the one I used from the 8.3 times of DOS, so my documents folder is DOC and I install the visual studio 2019 in the C:\dev\vs2019, never using spaces in names :)

I push a commit to fix the problem. See if it works.

With best regards

mhertz commented 4 years ago

I owe you several beers for all the help you have given me Danny! :) Thank you so much once again, really appreciate it, and was reported by affected user to work correctly now. Thanks again mate.

Febbe commented 2 years ago

I have 2 installations of msvc, the community 2019 and 2022 version, installed in the default paths and this occurrs. I could bypass it by enabling the "beta" of utf-8. The paths do not contain any ascents or special symbols. Only ASCII symbols.