Latest version breaks `poetry install` for certain packages

[mindflayer]

Hi there, I've just investigated a new failure appeared in my team's CI, discovering it's due to a new release which was just published (https://pypi.org/project/pyproject-hooks/1.1.0/).

Could you please have a look? We are using the latest versions of poetry and poetry-core, and pinning the previous version (1.0.0) of pyproject-hooks fixed it.

#5 43.30   Backend subprocess exited when trying to invoke get_requires_for_build_wheel
#5 43.30   
#5 43.30   Traceback (most recent call last):
#5 43.30     File "/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#5 43.30       main()
#5 43.30     File "/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#5 43.30       json_out['return_val'] = hook(**hook_input['kwargs'])
#5 43.30     File "/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
#5 43.30       backend = _build_backend()
#5 43.30     File "/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 74, in _build_backend
#5 43.30       ep = os.environ['PEP517_BUILD_BACKEND']
#5 43.30     File "/usr/lib/python3.10/os.py", line 680, in __getitem__
#5 43.30       raise KeyError(key) from None
#5 43.30   KeyError: 'PEP517_BUILD_BACKEND'
#5 43.30   
#5 43.30 
#5 43.30   at /.venv/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
#5 43.31       160│ 
#5 43.31       161│                 error = ChefBuildError("\n\n".join(message_parts))
#5 43.31       162│ 
#5 43.31       163│             if error is not None:
#5 43.31     → 164│                 raise error from None
#5 43.31       165│ 
#5 43.31       166│             return path
#5 43.31       167│ 
#5 43.31       168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
#5 43.31 
#5 43.31 Note: This error originates from the build backend, and is likely not a problem with poetry but with flask-basicauth (0.2.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "flask-basicauth (==0.2.0)"'.

[pradyunsg]

#5 43.30     File "/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 74, in _build_backend
#5 43.30       ep = os.environ['PEP517_BUILD_BACKEND']

This is not the code in 1.1.0: https://github.com/pypa/pyproject-hooks/blob/v1.1.0/src/pyproject_hooks/_in_process/_in_process.py#L74

Please provide more complete details for reproducing this issue.

[mindflayer]

I am trying to understand what happened and why pinning the version made it pass.

[mindflayer]

Our poetry.lock was pointing to pyproject-hooks==1.0.0 so it was downgraded during the packages installation. I am still trying to find the combination of tools that breaks the installation but it's definitely not an issue with the version released today.

I am still wondering why it happened today though...

Thanks for your support and sorry for the false alarm.

[igor47]

@mindflayer did you figure out? i'm having the same issue, also seems related to the pyproject-hooks update

[mdagost]

For me, I believe the root cause is that 1.0.0 was locked in the poetry.lock file, but then poetry itself pulled in 1.1.0 when it was pip installed. So there was a conflict. Either downgrading to 1.0.0 after pip installing, or upgrading to 1.1.0 in the lock file fixes things for me.

[dimbleby]

the environment variables passed from front-end to subprocess were changed at https://github.com/pypa/pyproject-hooks/pull/166

so my best guess as to how people are managing to hit this is:

• they start off in an environment that already contains poetry, and pyproject-hooks 1.0.0 (or 1.1.0)
• then they run poetry install or whatever
  • they have already gone wrong at this point, poetry should have been installed in its own isolated environment and not in the project they are managing
• as part of its work, poetry upgrades (or downgrades) pyproject-hooks to 1.1.0 (or 1.0.0)
• now poetry is still running the in-memory pyproject-hooks, but the pyproject-hooks that is available in the environment is the other version
• so next time poetry does something that uses a hook, it provides one set of environment variables but the subprocess executes code that expects the other set of environment variables
• and we fall over

[mindflayer]

@mindflayer did you figure out? i'm having the same issue, also seems related to the pyproject-hooks update

Sorry, I fixed pinning, as I mentioned yesterday, but I really had no time to spend in investigating the issue. Sorry.

As far as I understood so far, the problem is what @dimbleby explained. Thanks!

[takluyver]

That sounds like a solid hypothesis. :+1: Unfortunately I don't see much that we can do about it from this side - we'll obviously try not to change the child process interface more often than necessary, but it's meant to be an internal detail.

On Poetry's side, perhaps it could detect if it's targeting the environment it is running in, and either a) refuse to modify it at all, b) special case pyproject-hooks and refuse to change that, or c) special case pyproject-hooks and forcibly reload it after installing a new version.

[dimbleby]

I expect poetry will just tell people to read the docs and install it properly, I doubt code is needed.

[mindflayer]

I expect poetry will just tell people to read the docs and install it properly, I doubt code is needed.

Well, this is the advanced installation they list under their doc:

python3 -m venv $VENV_PATH
$VENV_PATH/bin/pip install -U pip setuptools
$VENV_PATH/bin/pip install poetry

It's not clear enough IMHO.

[mdagost]

Having gone through this process today, and having read the "big red box repeatedly", and noting that others are having this problem, perhaps a logical conclusion is that the poetry docs are not clear enough on the proper way to do installation, especially when docker is involved.