Closed matteosantama closed 1 year ago
Working with @matteosantama, we found it traces back to this method: https://github.com/python-poetry/poetry/blob/bf30ca696aa47bcc42937910d620c95c95c0741e/poetry/utils/env.py#L432
ETA: I directly tested the .get
method and it appears to be fine so it seems to be something with how that method ends up getting called and the virtual environment not being detected. It seems related to this issue I raised previously but not quite. https://github.com/python-poetry/poetry/issues/3054
I have not understood what is the issue here. Is it maybe a matter of calling poetry env use
to make sure the same environment is used afterwards?
Could you try to recreate and observe if anything changes in the content of ~/.cache/pypoetry/virtualenvs/envs.toml
in between runs?
Here's some further detail as I'm able to replicate and expand on the original issue
$ poetry --version
Poetry version 1.1.4
poetry build -vvv
:
$ poetry build -vvv
Using virtualenv: /Users/alexifm/.pyenv/versions/3.8.5/envs/infima3.8
Building infima (0.71.14)
- Building sdist
- Adding: /Users/alexifm/Projects/pyinfima/src/infima/__init__.py
- Adding: /Users/alexifm/Projects/pyinfima/src/infima/cli.py
- Adding: README.md
- Adding: pyproject.toml
- Built infima-0.71.14.tar.gz
- Building wheel
- Adding:
...
running build
running build_py
running build_ext
- Adding:
...
Skipping: /Users/alexifm/Projects/pyinfima/LICENSE
Skipping: /Users/alexifm/Projects/pyinfima/COPYING
- Built infima-0.71.14-cp39-cp39-macosx_10_15_x86_64.whl
Notice how a 3.9 wheel has been built even though it's supposed to be a package for 3.8 only. The fact that the build works also indicates that the correct virtual environment is being used since Cython and Numpy are needed for the build. It appears as though the correct virtualenv is used to build but the label for the wheel is using the Py version that is running Poetry (this is the only Python 3.9 I have on my system).
If I run poetry env use
, I get the following error:
$ poetry env use $(pyenv which python)
Skipping virtualenv creation, as specified in config file.
FileNotFoundError
[Errno 2] No such file or directory: '/Users/alexifm/Library/Caches/pypoetry/virtualenvs/envs.toml'
Okay, so I create the cache dir and rerun:
$ poetry env use $(pyenv which python)
Skipping virtualenv creation, as specified in config file.
Using virtualenv: /Users/alexifm/.local/pipx/venvs/poetry # <- Weird, this is not the virtualenv I specified with $(pyenv which python)
$ poetry build -vvv
The currently activated Python version 3.9.1 is not supported by the project (~3.8).
Trying to find and use a compatible version.
Trying python3
Using python3 (3.8.5)
Skipping virtualenv creation, as specified in config file.
Building infima (0.71.14)
- Building sdist
- Adding: /Users/alexifm/Projects/pyinfima/src/infima/__init__.py
- Adding: /Users/alexifm/Projects/pyinfima/src/infima/cli.py
- Adding: README.md
- Adding: pyproject.toml
- Built infima-0.71.14.tar.gz
- Building wheel
- Adding:
...
A setup.py file already exists. Using it.
Traceback (most recent call last):
File "/Users/alexifm/Projects/pyinfima/setup.py", line 201, in <module>
from build import *
File "/Users/alexifm/Projects/pyinfima/build.py", line 5, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
Stack trace:
9 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/clikit/console_application.py:131 in run
129│ parsed_args = resolved_command.args
130│
→ 131│ status_code = command.handle(parsed_args, io)
132│ except KeyboardInterrupt:
133│ status_code = 1
8 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/clikit/api/command/command.py:120 in handle
118│ def handle(self, args, io): # type: (Args, IO) -> int
119│ try:
→ 120│ status_code = self._do_handle(args, io)
121│ except KeyboardInterrupt:
122│ if io.is_debug():
7 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/clikit/api/command/command.py:171 in _do_handle
169│ handler_method = self._config.handler_method
170│
→ 171│ return getattr(handler, handler_method)(args, io, self)
172│
173│ def __repr__(self): # type: () -> str
6 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/cleo/commands/command.py:92 in wrap_handle
90│ self._command = command
91│
→ 92│ return self.handle()
93│
94│ def handle(self): # type: () -> Optional[int]
5 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/console/commands/build.py:36 in handle
34│
35│ builder = Builder(self.poetry)
→ 36│ builder.build(fmt, executable=self.env.python)
37│
4 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builder.py:30 in build
28│
29│ for builder in builders:
→ 30│ builder(self._poetry, executable=executable).build()
31│
3 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py:89 in build
87│ else:
88│ self._copy_module(zip_file)
→ 89│ self._build(zip_file)
90│
91│ self._write_metadata(zip_file)
2 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py:121 in _build
119│ try:
120│ os.chdir(str(self._path))
→ 121│ self._run_build_command(setup)
122│ finally:
123│ os.chdir(current_path)
1 ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py:149 in _run_build_command
147│
148│ def _run_build_command(self, setup):
→ 149│ subprocess.check_call(
150│ [
151│ self.executable.as_posix(),
CalledProcessError
Command '['/Users/alexifm/.local/pipx/venvs/poetry/bin/python', '/Users/alexifm/Projects/pyinfima/setup.py', 'build', '-b', '/Users/alexifm/Projects/pyinfima/build']' returned non-zero exit status 1.
at /usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py:373 in check_call
369│ if retcode:
370│ cmd = kwargs.get("args")
371│ if cmd is None:
372│ cmd = popenargs[0]
→ 373│ raise CalledProcessError(retcode, cmd)
374│ return 0
375│
376│
377│ def check_output(*popenargs, timeout=None, **kwargs):
envs.toml
file and install poetry with Python 3.8 using pipx. This will allow us to use the prevailing virtualenv to build the library instead of the one used to run Poetry and get the correct py version labeled on the wheel instead of the version used to run PoetryThanks for expanding. I see the issue now. I wouldn't know where to start, though.
Well it actually seems to be two issues. One is naming the wheel. The other is not using the correct environment when told to.
When I mentioned using poetry env use ...
I had not realized that you had deactivated virtualenvs.create
configuration setting. So I guess that could explain why poetry env use ...
does not work, or delivers unexpected results. I would say there is not point in using poetry env use ...
if the virtual environment creation is deactivated (my bad). Not sure how it should behave though, probably poetry should fail early with a meaningful message here.
You seem to say you use pyenv to create the virtual environment. As far as I know pyenv does not know how to do that. Do you mind clarifying that bit?
You seem to say you use pyenv to create the virtual environment. As far as I know pyenv does not know how to do that. Do you mind clarifying that bit?
We use this: https://github.com/pyenv/pyenv-virtualenv
So the command is $ pyenv virtualenv 3.8.5 test
When I mentioned using
poetry env use ...
I had not realized that you had deactivatedvirtualenvs.create
configuration setting. So I guess that could explain whypoetry env use ...
does not work, or delivers unexpected results. I would say there is not point in usingpoetry env use ...
if the virtual environment creation is deactivated (my bad). Not sure how it should behave though, probably poetry should fail early with a meaningful message here.
I'm not familiar with the env
command but it seems like it should still work. I read my configured setup as this: 1) Poetry shouldn't create venvs and instead should rely on the currently activated one, if possible; otherwise throw an error. 2) Poetry can override that and use some other venv, if told to with poetry env use
. This doesn't change the activated venv as it's just a Poetry thing.
In your case, it seems to me like there is no reason to deactivate the virtualenvs.create
configuration setting. If you activate your own custom virtual environment before calling any poetry command, then you should be good anyway. But if the setting is set to false
and you forget to activate a virtual environment, I think poetry might start installing things in the global environment.
Of course it still doesn't explain the issue though.
@sinoroc I opened a seperate issue regarding the naming of the wheel. Regarding poetry env
, that's a bit out of my depths as I just don't know that part of the project well enough. We can create a separate issue for that to isolate it and close out this issue.
I am out of my depth as well. If you feel confident that you have clearly identified 2 distinct issues, then yes feel free to close this one. As long as we have the link back to this current issue, it should be all good anyway.
I believe I am experiencing this issue as well on Arch Linux with Python 3.9:
This gist contains a minimal Python 3.8 pyproject.toml and the output of poetry install -vvv
The single dependency attempts to load a module in its setup.py, so at first I believed this to be an instance of #3501. However this case is different in that the dependency already contains a pyproject.toml with build-systems.requires defined.
I think the build fails because it is looking for a Python 3.9 wheel for tensorflow, which only supports up to Python 3.8. I installed poetry in a Python 3.8 docker container, and ran the build inside successfully.
Hey @cheesinglee The issue with sensenet is related to this comment: https://github.com/python-poetry/poetry/issues/3054#issuecomment-702912394
Basically, Poetry doesn't yet install build requirements but requires the build environment to be set up already. Our package handles this by adding whatever is required for build as a dev dependency. It's not ideal but Poetry is evolving and should end up improved on this front.
I can reproduce this issue on MacOS Big Sur as well.
@sinoroc https://gist.github.com/iserko/1e88e05675e360b23d52bac65c3baced replicated on Debian Buster (where poetry is installed using Homebrew and uses the Homebrew'd Python 3.9)
seems to be related to https://github.com/python-poetry/poetry/issues/3523
@iserko Thanks for all debugging and bug reporting efforts. It will help the maintainers when they come to this topic.
We ran into this on OSX. In our case we were telling poetry to not create a virtualenv but it insisted on writing to a virtualenv in the $HOME/Library/Caches/pypoetry/...
dir. We fixed this by blowing away the $HOME/Library/Caches/pypoetry
directory.
Using this workaround, it still works for us when we are using python 3.8 in our application and poetry is depending on python 3.10 as it does when installed from brew.
hth.
(search terms: work around, fix).
The original issue does not reproduce with virtualenvs.prefer-active-python true
on 1.2.x. There are some various other issues in the comments, but most related to wheels, incorrect use, and the virtualenvs.create
option being poorly designed/overloaded. I'm closing this for now -- if you have an issue expressed in the comments that reproduces on 1.2.x, please open a new issue.
i had this issue with 1.5.0 on my mac. Once the pypoetry dir existed in Caches, poetry insisted on using it. I didn't know there was a prefer-active-python config and lost several hours due to this.
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
OS: maOS 11.1 (Big Sur) Python: 3.8.5 (pyenv)
Poetry build is ignoring the virtual environment Python when building and instead using the Python it has been installed against.
When I install
poetry
withbrew
, it builds a Python 3.9 wheel and I getERROR: infima-0.71.9-cp39-cp39-macosx_11_1_x86_64.whl is not a supported wheel on this platform.
. When I installpoetry
withpipx
, it builds a wheel formacosx_10_16
(which does not exist), but I am able to install it.Steps to reproduce:
Then after
brew uninstall poetry
:Could be related to #3463