bug(encoding): UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 21: character maps to <undefined>

[Kristinita]

Another unicode issue. Related issues:

• bug(encoding): Don't modify files with non-ASCII filenames,
• UTF-8 in AppVeyor, Python 3.6.

1. Summary

For example, I have “citation marks” in my Pipfile.

If AppVeyor:

    I can't install dependencies for Pipenv.

else Travis CI or local machine:

    I successful install Pipenv dependencies.

2. Local environment

• Windows 10 Enterprise LTSB 64-bit EN,
• Python 3.6.4.

3. Project

See simply project configuration in SashaPipenv tree of my repository.

appveyor.yml file:

environment:

  matrix:

    - PYTHON: "C:\\Python36-x64"
      PYTHON_VERSION: "3.6.4"
      PYTHON_ARCH: "64"

platform: x64

init:
  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"

install:
  - pip install pipenv
  - pipenv install --dev

build_script:

  - pipenv --version
  - pelican --version

Pipfile:

# “Citation marks”

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

pelican = "*"

[requires]

python_version = "3.6"

In first line I have “quotation marks”.

4. Expected behavior

If local machine or same configuration for Travis CI:

    Dependencies successful installed.

Elif I remove first line from my Pipfile:

    Build passed

    (I have ImportError, but, possibly, it not related bug)

5. Actual behavior

Else AppVeyor:

Traceback (most recent call last):
File "C:\Python36-x64\Scripts\pipenv-script.py", line 11, in <module>
  load_entry_point('pipenv==11.1.3', 'console_scripts', 'pipenv')()
  File "c:\python36-x64\lib\site-packages\pipenv\vendor\click\core.py", line 722, in __call__
  return self.main(*args, **kwargs)
  File "c:\python36-x64\lib\site-packages\pipenv\vendor\click\core.py", line 697, in main
  rv = self.invoke(ctx)
  File "c:\python36-x64\lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\python36-x64\lib\site-packages\pipenv\vendor\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\python36-x64\lib\site-packages\pipenv\vendor\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\python36-x64\lib\site-packages\pipenv\cli.py", line 197, in install
    selective_upgrade=selective_upgrade
  File "c:\python36-x64\lib\site-packages\pipenv\core.py", line 1731, in do_install
    ensure_project(three=three, python=python, system=system, warn=True, deploy=deploy, skip_requiremen
      skip_requirements)
  File "c:\python36-x64\lib\site-packages\pipenv\core.py", line 622, in ensure_project
    ensure_virtualenv(three=three, python=python, site_packages=site_packages)
  File "c:\python36-x64\lib\site-packages\pipenv\core.py", line 567, in ensure_virtualenv
      python = ensure_python(three=three, python=python)
  File "c:\python36-x64\lib\site-packages\pipenv\core.py", line 443, in ensure_python
    python = project.required_python_version
  File "c:\python36-x64\lib\site-packages\pipenv\project.py", line 113, in required_python_version
      required = self.parsed_pipfile.get('requires', {}).get('python_full_version')
  File "c:\python36-x64\lib\site-packages\pipenv\project.py", line 255, in parsed_pipfile
      contents = f.read()
  File "c:\python36-x64\lib\encodings\cp1252.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 21: character maps to <undefined>
Command exited with code 1

6. Not helped

1. chcp 65001,
2. set PYTHONIOENCODING=utf-8,
3. chcp 65001 in chain of commands.

7. Do not offer

Please, do not offer, that I should not use non-ASCII symbols in Pipfile, because it works in another environments.

Thanks.

[Kristinita]

@IlyaFinkelshteyn , this relevant in May 2018.

Thanks.

[Kristinita]

The problem is relevant for May 2018.

Thanks.

[Kristinita]

The problem still exists. In the November, 2020, AppVeyor still doesn’t support Unicode 😿.

1. MCVE

Another example.

You can see this configuration in the KiraPeru branch of my repository for demo/debugging.

• appveyor.yml:

  platform: x64
  
  image: Visual Studio 2019
  
  clone_depth: 1
  
  environment:
    PYTHON: C:\Python39-x64
  
  init:
  - SETX PYTHONIOENCODING utf-8
  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
  
  install:
  - "%PYTHON%\\python -m pip install --upgrade pip"
  - "%PYTHON%\\python -m pip install peru"
  
  build_script:
  - peru sync

• peru.yaml:

  # “Quotes”
  git module open-graph:
    url: https://github.com/whiskyechobravo/pelican-open_graph/

2. Behavior

2.1. Current

• AppVeyor build:

  peru sync
  Traceback (most recent call last):
    File "C:\Python39-x64\Scripts\peru-script.py", line 12, in <module>
      sys.exit(main())
    File "C:\Python39-x64\lib\site-packages\peru\main.py", line 366, in main
      parser.warn_duplicate_keys(runtime.peru_file)
    File "C:\Python39-x64\lib\site-packages\peru\parser.py", line 249, in warn_duplicate_keys
      text = f.read()
    File "C:\Python39-x64\lib\encodings\cp1252.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
  UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 13: character maps to <undefined>
  Command exited with code 1

2.2. Expected

If in peru.yaml:

- # “Quotes”
+ # Without quotes

• AppVeyor build:

  peru sync
  Discovering tests…OK
  Build success

Thanks.

[FeodorFitsner]

It's not AppVeyor issue. The problem is in peru.yaml file having Unicode quotes.

I can reproduce the issue locally:

git clone --branch=KiraPeru https://gitlab.com/Kristinita/kirapipenv.git
cd kirapipenv
git checkout 42411cc1
python
>>> f = open('peru.yaml')
>>> r = f.read()

My rough guess is peru.yaml not having BOM header: