Special characters not supported in the authors section of pyproject.toml

[taljaards]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 17763.253

• Poetry version: 0.12.11

• pyproject.toml file: https://gist.github.com/taljaards/b06c1dda1ece0b8d443eb9512213e47d

Issue

Poetry does not support special characters in the "authors" section of the pyproject.toml file. I created a new folder, did poetry init and answered no to most of the yes/no questions.

Afterwards, when doing poetry add, it fails:

C:\Users\Stephan\Source\jupyter-lab>poetry add jupyterlab -vvv

[UnicodeDecodeError]
'utf-8' codec can't decode byte 0xe9 in position 88: invalid continuation byte

Exception trace:
 C:\Users\Stephan\.poetry\lib\poetry\_vendor\py3.7\cleo\application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 C:\Users\Stephan\.poetry\lib\poetry\console\application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 C:\Users\Stephan\.poetry\lib\poetry\_vendor\py3.7\cleo\application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 C:\Users\Stephan\.poetry\lib\poetry\console\commands\command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 C:\Users\Stephan\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\base_command.py in run() at line 136
   self.initialize(input_, output_)
 C:\Users\Stephan\.poetry\lib\poetry\console\commands\env_command.py in initialize() at line 18
   current_env = Env.get(self.poetry.file.parent)
 C:\Users\Stephan\.poetry\lib\poetry\console\commands\command.py in poetry() at line 62
   return self.get_application().poetry
 C:\Users\Stephan\.poetry\lib\poetry\console\application.py in poetry() at line 60
   self._poetry = Poetry.create(os.getcwd())
 C:\Users\Stephan\.poetry\lib\poetry\poetry.py in create() at line 98
   local_config = TomlFile(poetry_file.as_posix()).read()
 C:\Users\Stephan\.poetry\lib\poetry\_vendor\py3.7\tomlkit\toml_file.py in read() at line 20
   return loads(f.read())
 C:\Users\Stephan\AppData\Local\Programs\Python\Python37\lib\codecs.py in decode() at line 322
   (result, consumed) = self._buffer_decode(data, self.errors, final)

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

This is because of the "é" character in my name. After removing the "é", operations using the pyproject.toml file work as expected.

[digitalresistor]

Are you writing the pyproject.toml in UTF-8? Are you using a UTF-8 character for the é?

[sdispater]

I had this issue too when using init since it uses the output of git config --list to populate the default authors section but the é in Git was not properly encoded. Resetting it with git config solved the issue for me.

[nuno-andre]

Another é-named here.

I think the problem (also for #221) is that subprocess calls git pipelining through Windows stdio, which hasn't native support for UTF-8, the MinGW default encoding. Hence the replacement character.

From Python 3.7 (PEP 540) this can be circumvented by setting the environment variable PYTHONUTF8 which makes Python to ignore local encoding using UTF-8.

So, this annoyance for us the beyondasciians (sorry) can be avoided in cmd with:

set PYTHONUTF8=1 && poetry init

Or for posh:

$env:PYTHONUTF8=1; poetry init

[vlcinsky]

I would say, the problem is related to writing pyproject.toml file without explicit utf-8 encoding thus using default, what is likely to fail on some Windows.

https://github.com/sdispater/poetry/blob/master/poetry/console/commands/init.py#L161

[vlcinsky]

Fixed in PR #1085 or in it's extended version (fixing more places dealing with properly stating encoding to use) PR #1087

[vlcinsky]

@taljaards can you check, if the latest poetry 0.12.17 works well?

[taljaards]

@vlcinsky Yup, it works! Thanks.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.