"Managing environments" confusing documentation about honoring current virtual environment

[jfly]

• [X] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

The Poetry docs on "Managing environments" opens with this simple statement:

... it [Poetry] will first check if it’s currently running inside a virtual environment. If it is, it will use it directly without creating a new one.

This doesn't seem to be entirely true, as poetry env use seems to take precedence over the current virtual environment.

Proof

Set up new directory with fresh venv: $ mkdir poetry-venv-confusion-demo $ cd poetry-venv-confusion-demo Init poetry + see that it does not see a venv: $ poetry init -n $ poetry env info Virtualenv Python: 3.10.12 Implementation: CPython Path: NA Executable: NA ... Do something that forces poetry to create a venv: $ poetry install Creating virtualenv poetry-venv-confusion-demo-TAVwgqF2-py3.10 in /home/jeremy/.cache/pypoetry/virtualenvs ... Now poetry sees that venv it just created: $ poetry env info Virtualenv Python: 3.10.12 Implementation: CPython Path: /home/jeremy/.cache/pypoetry/virtualenvs/poetry-venv-confusion-demo-TAVwgqF2-py3.10 Executable: /home/jeremy/.cache/pypoetry/virtualenvs/poetry-venv-confusion-demo-TAVwgqF2-py3.10/bin/python Valid: True Create + activate venv + proof it's activated: $ python -m venv .my-venv # poetry has special handling of the .venv directory, so choosing something else here $ source .my-venv/bin/activate $ which python /home/jeremy/tmp/poetry-venv-confusion-demo/.my-venv/bin/python $ echo $VIRTUAL_ENV /home/jeremy/tmp/poetry-venv-confusion-demo/.my-venv See that poetry honors the current active venv: $ poetry env info Virtualenv Python: 3.10.12 Implementation: CPython Path: /home/jeremy/tmp/poetry-venv-confusion-demo/.my-venv Executable: /home/jeremy/tmp/poetry-venv-confusion-demo/.my-venv/bin/python Valid: True ... So far so good. As promised, poetry is preferring the active venv (in `.my-venv`) over the one it created in `~/.cache//pypoetry/virtualenvs` earlier. Here's the rub. If I run `poetry env use`, then poetry **stops honoring my active venv**: $ poetry env use python3.10 Using virtualenv: /home/jeremy/.cache/pypoetry/virtualenvs/poetry-venv-confusion-demo-TAVwgqF2-py3.10 $ poetry env info Virtualenv Python: 3.10.12 Implementation: CPython Path: /home/jeremy/.cache/pypoetry/virtualenvs/poetry-venv-confusion-demo-TAVwgqF2-py3.10 Executable: /home/jeremy/.cache/pypoetry/virtualenvs/poetry-venv-confusion-demo-TAVwgqF2-py3.10/bin/python Valid: True

Here's the relevant code: https://github.com/python-poetry/poetry/blob/1.5.1/src/poetry/utils/env.py#L680-L684. Running poetry env use creates a ~/.cache/pypoetry/virtualenvs/envs.toml (the same as the TOMLFile(venv_path / self.ENVS_FILE) you see in the code) that takes precedence over the fact that we're currently in a venv.

I can see arguments for poetry choosing to behave this way. It's easy to imagine users being surprised if running poetry env use python3.10 had no effect because they're actually already in a venv. That said, the current behavior is extremely confusing for people who want to bring their own venv management. If the user ever ran poetry env use, they're going to be pretty surprised to find that poetry has no interest in the venv that they've so carefully activated.

More context: I am on a team of ~40 engineers at a python shop. We use direnv to manage venvs for our projects, and we really just want poetry to be a tool for managing python deps, we do not want it to manage venvs. I ended up in a bit of a marathon debugging session earlier this week because poetry wasn't installing deps into a developer's venv, and it took us forever to discover that this was because he had a envs.toml file in his poetry cache dir (presumably because he ran poetry env use at some distant point in the past).

Would y'all be open to changing poetry to make these sorts of environment misconfigs less likely to happen or less confusing? Some ideas:

• Have poetry complain (perhaps even abort) with a clear error/warning message if it finds that there's both a current venv and an entry in the envs.toml file for the current directory.
• Add a config setting to put poetry in a "do not do any venv management" mode
• Or, if neither of the above are palatable, change the "Managing environments" docs to explain that poetry prefers poetry use environments over a currently active venv.

[mbp101]

Thank you @jfly for your research and surfacing this issue.

I just spent 8+ hours over the course of 2 days digging into a very similar issue.

After doing a complete de-installation of poetry, pyenv, pyenv-virtualenv, and HomeBrew python version 3.11.5 and then re-installing and re-configuring everything from scratch, Poetry continued to refuse to detect the virtualenv generated by pyenv !

Like you, I was forced into experiencing my own marathon debugging session.

It turned out that there was an envs.toml file located at ~/Library/Caches/pypoetry/virtualenvs/envs.toml that was the culprit. At some time in the past, I must have blindly executed a poetry env use command w/o understanding the consequences it would have.

What surprised me is that the files in this cache seemed to persist even after a Poetry uninstall.

So I would humbly suggest an addition to your recommendations above:

• Ensure Poetry properly cleans all caches on uninstall or update the docs to indicate that cached files continue to linger after an uninstall and may be worth considering when debugging issues.