UnicodeDecodeError .. aka 'terminal encoding problem problem '

[dubiouscript]

just ftr

adding the

export LC_CTYPE=en_US.UTF-8

sloution from : https://stackoverflow.com/a/49127686 ;]

to my env

fixes this "UnicodeDecodeError" !

  File "./python-package-init.py", line 240, in <module>
    main()
  File "./python-package-init.py", line 27, in main
    metadata = package_json_to_metadata(data, args.package, args.version)
  File "./python-package-init.py", line 95, in package_json_to_metadata
    metadata.update(determine_package_dependencies(package_json, metadata['url']))
  File "./python-package-init.py", line 161, in determine_package_dependencies
    dependencies = determine_dependencies_from_package(url)
  File "./python-package-init.py", line 176, in determine_dependencies_from_package
    import setup  # This is setup.py which calls setuptools.setup
  File "./setup.py", line 18, in <module>
  File "./setup.py", line 6, in read
  File "/nix/store/1c4zshbxizcsvyj46fwf5jj10h41wf6i-python3-3.6.6/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1691: ordinal not in range(128)

[costrouc]

Thanks your for that! You actually answered an issue that I was having in a PR that I had in nixos/nixpkgs. https://github.com/NixOS/nixpkgs/pull/48171#discussion_r224953036

[dubiouscript]

no problem ... can this var be passed somehow with nix-shell ?.. args ?!

#! /usr/bin/env nix-shell
#! nix-shell -i python3 -p python3 python36Packages.jinja2 python36Packages.setuptools

i looked a little but did not find any obvious method ..

[costrouc]

I am not aware of how this can be done with shebangs. But this might work. Could you try adding this to the script? Sadly I can't reproduce this error but if it works I will add it to the script

#! /usr/bin/env nix-shell
#! nix-shell -i python3 -p python3 python36Packages.jinja2 python36Packages.setuptools

import os
os.environ['LC_ALL'] = "en_US.UTF-8"

[dubiouscript]

thanks! .. digging around the net i found that aparently 'LANG and LC_CTYPE env variables cause many issues with unicode chars' - https://github.com/zeit/hyper/issues/607

https://stackoverflow.com/questions/30479607/explain-the-effects-of-export-lang-lc-ctype-lc-all

i encountered the problem runing python-package-init.py with nix 2.1.2 on lubuntu 18.10 x86_64 beta "live/squashfs"

unfortunatly now i can't reproduce the error in a new terminal =/ ..ill try to reproduce again after rebooting

[dubiouscript]

not rebooted yet but

i can now produce the above error after making a few changes eg

-#! nix-shell -i python3 -p python3 python36Packages.jinja2 python36Packages.setuptools 
+#! nix-shell -i python3 -p python3 python36Packages.jinja2 python36Packages.setuptools --pure 

-    stdout = subprocess.check_output(['nix-prefetch-url', '--unpack', url], stderr=subprocess.STDOUT)
+    stdout = subprocess.check_output([[os.environ["HOME"]+'/.nix-profile/bin/nix-prefetch-url', '--unpack', url], stderr=subprocess.STDOUT)

however exporting LC_CTYPE or adding os.environ['LC_ALL']

now do not appear to make any diffrence 0.o

i also tried adding '--argstr "LC_CTYPE" "en_US.UTF-8" ' to nix shell this also appears to make no diffrence

perhaps relevent? https://github.com/NixOS/nix/issues/599 'Nix broken by having LANG and LC_* set'