poetry env use fails with: ModuleNotFoundError: No module named 'encodings'

[dataspun]

• Poetry version: 1.2.2

• Python version: 3.10.8

• OS version and name: Windows 11

• pyproject.toml: n/a

• [x ] I am on the latest stable Poetry version, installed using a recommended method.

• [ x] I have searched the issues of this repo and believe that this is not a duplicate.

• [ x] I have consulted the FAQ and blog for any relevant entries or release notes.

• [x ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Having installed a standard, Python.org Python as well as poetry using official Powershell script, I attempt to use a separate python environment:

poetry env use /full/path/to/python

Here is the output:

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Attempting poetry debug info generates the same error.

[neersighted]

You have not attached the verbose output -- please run the command again with -vvv. Also, please let us know where your Python was installed from (Python.org installer, Microsoft Store, Anaconda/miniconda, ???).

[dataspun]

@neersighted latest Python.org Python 3.10.8

[dataspun]

@neersighted the -vvv output:

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/full/path/to/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/full/path/to/python/bin/python3.exe'
  sys.base_prefix = ''
  sys.base_exec_prefix = ''
  sys.platlibdir = 'lib'
  sys.executable = '/full/path/to/python/bin/python3.exe'
  sys.prefix = ''
  sys.exec_prefix = ''
  sys.path = [
    '/full/path/to/python/bin/python39.zip',
    '.\\DLLs',
    '.\\lib',
    '/full/path/to/python/bin',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00004dfc (most recent call first):
<no Python frame>

  Stack trace:

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:523 in _full_python_path
       521│         try:
       522│             executable = decode(
    →  523│                 subprocess.check_output(
       524│                     list_to_shell_command(
       525│                         [python, "-c", '"import sys; print(sys.executable)"']

  1  ~\AppData\Local\Programs\Python\Python310\lib\subprocess.py:421 in check_output
       419│         kwargs['input'] = empty
       420│
    →  421│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       422│                **kwargs).stdout
       423│

  CalledProcessError

  Command '"/full/path/to/python" -c "import sys; print(sys.executable)"' returned non-zero exit status 1.

  at ~\AppData\Local\Programs\Python\Python310\lib\subprocess.py:526 in run
       522│             # We don't call process.wait() as .__exit__ does that for us.
       523│             raise
       524│         retcode = process.poll()
       525│         if check and retcode:
    →  526│             raise CalledProcessError(retcode, process.args,
       527│                                      output=stdout, stderr=stderr)
       528│     return CompletedProcess(process.args, retcode, stdout, stderr)
       529│
       530│

The following error occurred when trying to handle this error:

  Stack trace:

  9  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:329 in run
      327│
      328│             try:
    → 329│                 exit_code = self._run(io)
      330│             except Exception as e:
      331│                 if not self._catch_exceptions:

  8  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:185 in _run
      183│         self._load_plugins(io)
      184│
    → 185│         exit_code: int = super()._run(io)
      186│         return exit_code
      187│

  7  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:423 in _run
      421│             io.input.set_stream(stream)
      422│
    → 423│         exit_code = self._run_command(command, io)
      424│         self._running_command = None
      425│

  6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:465 in _run_command
      463│
      464│         if error is not None:
    → 465│             raise error
      466│
      467│         return event.exit_code

  5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:449 in _run_command
      447│
      448│             if event.command_should_run():
    → 449│                 exit_code = command.run(io)
      450│             else:
      451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\base_command.py:119 in run
      117│         io.input.validate()
      118│
    → 119│         status_code = self.execute(io)
      120│
      121│         if status_code is None:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\command.py:83 in execute
       81│
       82│         try:
    →  83│             return self.handle()
       84│         except KeyboardInterrupt:
       85│             return 1

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\env\use.py:24 in handle
       22│             return 0
       23│
    →  24│         env = manager.activate(self.argument("python"), self.io)
       25│
       26│         self.line(f"Using virtualenv: {env.path}")

  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:569 in activate
       567│             pass
       568│
    →  569│         python = self._full_python_path(python)
       570│
       571│         try:

  EnvCommandError

  Command "/full/path/to/python" -c "import sys; print(sys.executable)" errored with the following return code 1, and output:

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:531 in _full_python_path
       527│                     shell=True,
       528│                 ).strip()
       529│             )
       530│         except CalledProcessError as e:
    →  531│             raise EnvCommandError(e)
       532│
       533│         return executable
       534│
       535│     def _detect_active_python(self, io: IO) -> str | None:

[neersighted]

Seems like an issue with your Python install, and PYTHONPATH/PYTHONHOME being unset. I'd suggest reinstalling Python and choosing the "Add Python to environment variables" option.

[dataspun]

@neersighted thanks, I checked that box during Python install.

Again, it just seems like there's something missing, at the very least, from the poetry installation docs. Possibly even the poetry installer for Windows.

[neersighted]

Do things work as expected for you now?

I don't think installing a well-behaved Python is within scope for Poetry. We can add FAQ entries if we determine some of the issues you ran into are truly common -- however, they don't appear to be. Your first issue is Anaconda DLL hell, which is not something we can really anticipate, though we could add a general FAQ entry for SSL not working. Your second issue here appears to be a poorly cleaned up Anaconda install requiring you to explicitly ask the Python installer to override the necessary environment variables. This is such an obscure situation and so specific to your local installation that I do not thing there is anything Poetry can do.

In general, you are just running into the state of Python on Windows. Poetry merely requires a well-behaved Python install and doesn't do or require anything special on Windows. PRs are welcome, but I personally would want to see more evidence this is common before we try to head it off at the Poetry level.

[dataspun]

@neersighted no, things don't work as expected. To clear up any confusion from above: I checked the "Add Python to environment variables" option during the standard, Python.org install (and re-install) but it doesn't prevent (or fix) the issue with poetry. In all other respects, Python is well-behaved, it's just that poetry doesn't work. Mentioning these issues (and proposed workarounds) in the installation docs or general FAQ certainly seems appropriate. They're not obscure, they're unavoidable and unacknowledged.

[neersighted]

To further debug this, please run the command Poetry is using to introspect your Python interactively, e.g. /full/path/to/python -c "import sys; print(sys.executable)". We can then determine if something Poetry is doing to the environment image is causing the issue, or if the Python itself is the issue.

Regarding "issues," you are the first user to run into both of the issues you've opened today. These are new and I (nor any other regular contributors) do not have solutions for them yet, as they are novel to us. If they truly are revealed to be common gotchas, we can certainly discuss adding FAQ entries. As things stand, Poetry is installed over a hundred thousand times a month on Windows and we have yet to see anybody reach about about the difficulties you are having.

[dataspun]

The output of your suggested statement:

/full/path/to/python/bin/python3.exe

In other words, it returns the name of the Python executable in the path I provided. Which is the same as printed above in the -vvv output. Thanks again for providing responsive support on this issue!

[neersighted]

Hmm, that's reassuring, but that is not what your above stacktrace shows. It shows an exit with an error code, and error output reporting the missing encodings module which prevents importing sys. I'll have to review how env.py manipulates the environment (I know we don't pass it through wholesale).

In the meantime, it would be incredibly useful if you could reproduce this in a Windows container. Until it can be reproduced reliably, we'll be stuck either guessing and giving you things to check, or waiting for you to develop expertise in Python on Windows so that you can diagnose it yourself.

[neersighted]

(also, hopefully it goes without saying that you should at the very least re-log, if not reboot, after doing the installs/uninstalls from earlier, incase an old variable is sticking around. PYTHONHOME should be empty -- maybe check your system environment variables slash print $env:PYTHONHOME in a fresh PowerShell session after reboot/relog to confirm that.)

[dataspun]

@neersighted: PYTHONHOME and PYTHONPATH are empty

[Corfucinas]

I also ran into this issue, I uninstalled python 3.10.8 and installed 3.11.0 using pyenv and setting it as the global interpreter. Just decided to uninstall poetry and reinstall it using the recommended method and it's working again fine.

[neersighted]

On what platform? Where was Python 3.10.8 installed from?

[cieske]

In my case I removed conda env which is set as sys.path, sys.exec_prefix, etc... and encountered same error. After re-create the env with same name, everything works fine.