Poetry build is ignoring the virtual environment Python when building and instead using the Python it has been installed against

[matteosantama]

OS: maOS 11.1 (Big Sur) Python: 3.8.5 (pyenv)

Poetry build is ignoring the virtual environment Python when building and instead using the Python it has been installed against.

When I install poetry with brew, it builds a Python 3.9 wheel and I get ERROR: infima-0.71.9-cp39-cp39-macosx_11_1_x86_64.whl is not a supported wheel on this platform.. When I install poetry with pipx, it builds a wheel for macosx_10_16 (which does not exist), but I am able to install it.

Steps to reproduce:

$ brew install poetry
$ poetry build
Building infima (0.71.9)
  - Building sdist
  - Built infima-0.71.9.tar.gz
  - Building wheel
running build
running build_py
package init file 'src/infima/datasets/embs/sec/sql/__init__.py' not found (or not a regular file)
running build_ext
  - Built infima-0.71.9-cp39-cp39-macosx_11_1_x86_64.whl

Then after brew uninstall poetry:

$ pipx install poetry --python python3.8
$ poetry build
Building infima (0.71.9)
  - Building sdist
  - Built infima-0.71.9.tar.gz
  - Building wheel
running build
running build_py
package init file 'src/infima/datasets/embs/sec/sql/__init__.py' not found (or not a regular file)
running build_ext
  - Built infima-0.71.9-cp38-cp38-macosx_10_16_x86_64.whl

$ poetry config --list
cache-dir = "/Users/matteosantamaria/Library/Caches/pypoetry"
experimental.new-installer = true
installer.parallel = true
repositories.gitlab.url = "https://gitlab.com/..."
virtualenvs.create = false
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/matteosantamaria/Library/Caches/pypoetry/virtualenvs

Could be related to #3463

[alexifm]

Working with @matteosantama, we found it traces back to this method: https://github.com/python-poetry/poetry/blob/bf30ca696aa47bcc42937910d620c95c95c0741e/poetry/utils/env.py#L432

ETA: I directly tested the .get method and it appears to be fine so it seems to be something with how that method ends up getting called and the virtual environment not being detected. It seems related to this issue I raised previously but not quite. https://github.com/python-poetry/poetry/issues/3054

[sinoroc]

I have not understood what is the issue here. Is it maybe a matter of calling poetry env use to make sure the same environment is used afterwards?

Could you try to recreate and observe if anything changes in the content of ~/.cache/pypoetry/virtualenvs/envs.toml in between runs?

[alexifm]

Here's some further detail as I'm able to replicate and expand on the original issue

0. Version:

   $ poetry --version
   Poetry version 1.1.4

1. We have a build script that compiles Cython and are restricting everything to Python 3.8. We have a virtualenv set for the project directory using pyenv. So far, everything runs fine against this virtualenv.
2. Here is the result of poetry build -vvv:

   $ poetry build -vvv
   Using virtualenv: /Users/alexifm/.pyenv/versions/3.8.5/envs/infima3.8
   Building infima (0.71.14)
   - Building sdist
   - Adding: /Users/alexifm/Projects/pyinfima/src/infima/__init__.py
   - Adding: /Users/alexifm/Projects/pyinfima/src/infima/cli.py
   - Adding: README.md
   - Adding: pyproject.toml
   - Built infima-0.71.14.tar.gz
   - Building wheel
   - Adding: 
   ...
   running build
   running build_py
   running build_ext
   - Adding: 
   ...
   Skipping: /Users/alexifm/Projects/pyinfima/LICENSE
   Skipping: /Users/alexifm/Projects/pyinfima/COPYING
   - Built infima-0.71.14-cp39-cp39-macosx_10_15_x86_64.whl

   Notice how a 3.9 wheel has been built even though it's supposed to be a package for 3.8 only. The fact that the build works also indicates that the correct virtual environment is being used since Cython and Numpy are needed for the build. It appears as though the correct virtualenv is used to build but the label for the wheel is using the Py version that is running Poetry (this is the only Python 3.9 I have on my system).

3. If I run poetry env use, I get the following error:

   $ poetry env use $(pyenv which python)
   Skipping virtualenv creation, as specified in config file.
   
   FileNotFoundError
   
   [Errno 2] No such file or directory: '/Users/alexifm/Library/Caches/pypoetry/virtualenvs/envs.toml'

4. Okay, so I create the cache dir and rerun:

   $ poetry env use $(pyenv which python)                        
   Skipping virtualenv creation, as specified in config file.
   Using virtualenv: /Users/alexifm/.local/pipx/venvs/poetry     # <- Weird, this is not the virtualenv I specified with $(pyenv which python)
   $ poetry build -vvv 
   The currently activated Python version 3.9.1 is not supported by the project (~3.8).
   Trying to find and use a compatible version. 
   Trying python3
   Using python3 (3.8.5)
   Skipping virtualenv creation, as specified in config file.
   Building infima (0.71.14)
   - Building sdist
   - Adding: /Users/alexifm/Projects/pyinfima/src/infima/__init__.py
   - Adding: /Users/alexifm/Projects/pyinfima/src/infima/cli.py
   - Adding: README.md
   - Adding: pyproject.toml
   - Built infima-0.71.14.tar.gz
   - Building wheel
   - Adding: 
   ...
   A setup.py file already exists. Using it.
   Traceback (most recent call last):
   File "/Users/alexifm/Projects/pyinfima/setup.py", line 201, in <module>
   from build import *
   File "/Users/alexifm/Projects/pyinfima/build.py", line 5, in <module>
   import numpy as np
   ModuleNotFoundError: No module named 'numpy'
   
   Stack trace:
   
   9  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/clikit/console_application.py:131 in run
     129│             parsed_args = resolved_command.args
     130│ 
   → 131│             status_code = command.handle(parsed_args, io)
     132│         except KeyboardInterrupt:
     133│             status_code = 1
   
   8  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/clikit/api/command/command.py:120 in handle
     118│     def handle(self, args, io):  # type: (Args, IO) -> int
     119│         try:
   → 120│             status_code = self._do_handle(args, io)
     121│         except KeyboardInterrupt:
     122│             if io.is_debug():
   
   7  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/clikit/api/command/command.py:171 in _do_handle
     169│         handler_method = self._config.handler_method
     170│ 
   → 171│         return getattr(handler, handler_method)(args, io, self)
     172│ 
     173│     def __repr__(self):  # type: () -> str
   
   6  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/cleo/commands/command.py:92 in wrap_handle
      90│         self._command = command
      91│ 
   →  92│         return self.handle()
      93│ 
      94│     def handle(self):  # type: () -> Optional[int]
   
   5  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/console/commands/build.py:36 in handle
     34│ 
     35│         builder = Builder(self.poetry)
   → 36│         builder.build(fmt, executable=self.env.python)
     37│ 
   
   4  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builder.py:30 in build
     28│ 
     29│         for builder in builders:
   → 30│             builder(self._poetry, executable=executable).build()
     31│ 
   
   3  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py:89 in build
      87│             else:
      88│                 self._copy_module(zip_file)
   →  89│                 self._build(zip_file)
      90│ 
      91│             self._write_metadata(zip_file)
   
   2  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py:121 in _build
     119│                     try:
     120│                         os.chdir(str(self._path))
   → 121│                         self._run_build_command(setup)
     122│                     finally:
     123│                         os.chdir(current_path)
   
   1  ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py:149 in _run_build_command
     147│ 
     148│     def _run_build_command(self, setup):
   → 149│         subprocess.check_call(
     150│             [
     151│                 self.executable.as_posix(),
   
   CalledProcessError
   
   Command '['/Users/alexifm/.local/pipx/venvs/poetry/bin/python', '/Users/alexifm/Projects/pyinfima/setup.py', 'build', '-b', '/Users/alexifm/Projects/pyinfima/build']' returned non-zero exit status 1.
   
   at /usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py:373 in check_call
      369│     if retcode:
      370│         cmd = kwargs.get("args")
      371│         if cmd is None:
      372│             cmd = popenargs[0]
   →  373│         raise CalledProcessError(retcode, cmd)
      374│     return 0
      375│ 
      376│ 
      377│ def check_output(*popenargs, timeout=None, **kwargs):

5. To make this work, I need to remove the envs.toml file and install poetry with Python 3.8 using pipx. This will allow us to use the prevailing virtualenv to build the library instead of the one used to run Poetry and get the correct py version labeled on the wheel instead of the version used to run Poetry

[sinoroc]

Thanks for expanding. I see the issue now. I wouldn't know where to start, though.

[alexifm]

Well it actually seems to be two issues. One is naming the wheel. The other is not using the correct environment when told to.

[sinoroc]

When I mentioned using poetry env use ... I had not realized that you had deactivated virtualenvs.create configuration setting. So I guess that could explain why poetry env use ... does not work, or delivers unexpected results. I would say there is not point in using poetry env use ... if the virtual environment creation is deactivated (my bad). Not sure how it should behave though, probably poetry should fail early with a meaningful message here.

You seem to say you use pyenv to create the virtual environment. As far as I know pyenv does not know how to do that. Do you mind clarifying that bit?

[alexifm]

You seem to say you use pyenv to create the virtual environment. As far as I know pyenv does not know how to do that. Do you mind clarifying that bit?

We use this: https://github.com/pyenv/pyenv-virtualenv So the command is $ pyenv virtualenv 3.8.5 test

[alexifm]

When I mentioned using poetry env use ... I had not realized that you had deactivated virtualenvs.create configuration setting. So I guess that could explain why poetry env use ... does not work, or delivers unexpected results. I would say there is not point in using poetry env use ... if the virtual environment creation is deactivated (my bad). Not sure how it should behave though, probably poetry should fail early with a meaningful message here.

I'm not familiar with the env command but it seems like it should still work. I read my configured setup as this: 1) Poetry shouldn't create venvs and instead should rely on the currently activated one, if possible; otherwise throw an error. 2) Poetry can override that and use some other venv, if told to with poetry env use. This doesn't change the activated venv as it's just a Poetry thing.

[sinoroc]

In your case, it seems to me like there is no reason to deactivate the virtualenvs.create configuration setting. If you activate your own custom virtual environment before calling any poetry command, then you should be good anyway. But if the setting is set to false and you forget to activate a virtual environment, I think poetry might start installing things in the global environment.

Of course it still doesn't explain the issue though.

[alexifm]

@sinoroc I opened a seperate issue regarding the naming of the wheel. Regarding poetry env, that's a bit out of my depths as I just don't know that part of the project well enough. We can create a separate issue for that to isolate it and close out this issue.

[sinoroc]

I am out of my depth as well. If you feel confident that you have clearly identified 2 distinct issues, then yes feel free to close this one. As long as we have the link back to this current issue, it should be all good anyway.

[cheesinglee]

I believe I am experiencing this issue as well on Arch Linux with Python 3.9:

This gist contains a minimal Python 3.8 pyproject.toml and the output of poetry install -vvv

The single dependency attempts to load a module in its setup.py, so at first I believed this to be an instance of #3501. However this case is different in that the dependency already contains a pyproject.toml with build-systems.requires defined.

I think the build fails because it is looking for a Python 3.9 wheel for tensorflow, which only supports up to Python 3.8. I installed poetry in a Python 3.8 docker container, and ran the build inside successfully.

[alexifm]

Hey @cheesinglee The issue with sensenet is related to this comment: https://github.com/python-poetry/poetry/issues/3054#issuecomment-702912394

Basically, Poetry doesn't yet install build requirements but requires the build environment to be set up already. Our package handles this by adding whatever is required for build as a dev dependency. It's not ideal but Poetry is evolving and should end up improved on this front.

[rushilsrivastava]

I can reproduce this issue on MacOS Big Sur as well.

[iserko]

@sinoroc https://gist.github.com/iserko/1e88e05675e360b23d52bac65c3baced replicated on Debian Buster (where poetry is installed using Homebrew and uses the Homebrew'd Python 3.9)

seems to be related to https://github.com/python-poetry/poetry/issues/3523

[sinoroc]

@iserko Thanks for all debugging and bug reporting efforts. It will help the maintainers when they come to this topic.

[ehiggs]

We ran into this on OSX. In our case we were telling poetry to not create a virtualenv but it insisted on writing to a virtualenv in the $HOME/Library/Caches/pypoetry/... dir. We fixed this by blowing away the $HOME/Library/Caches/pypoetry directory.

Using this workaround, it still works for us when we are using python 3.8 in our application and poetry is depending on python 3.10 as it does when installed from brew.

hth.

(search terms: work around, fix).

[neersighted]

The original issue does not reproduce with virtualenvs.prefer-active-python true on 1.2.x. There are some various other issues in the comments, but most related to wheels, incorrect use, and the virtualenvs.create option being poorly designed/overloaded. I'm closing this for now -- if you have an issue expressed in the comments that reproduces on 1.2.x, please open a new issue.

[michaelmattox-honor]

i had this issue with 1.5.0 on my mac. Once the pypoetry dir existed in Caches, poetry insisted on using it. I didn't know there was a prefer-active-python config and lost several hours due to this.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.