Closed alexdewar closed 1 year ago
I've dug in a bit more and it's weirder than I thought. This following pyproject.toml doesn't work:
[project]
name = "SolidityGUI"
version = "0.1"
authors = ["author1"]
dependencies = ["pygmsh", "pyside6", "vtk"]
[project.optional-dependencies]
dev = ["black"]
However, if you remove the authors
line, it works fine again:
[project]
name = "SolidityGUI"
version = "0.1"
dependencies = ["pygmsh", "pyside6", "vtk"]
[project.optional-dependencies]
dev = ["black"]
How odd is that?!
Thanks! Can you post the entire pyproject.toml
, including the build-system
section?
That is the entire pyproject.toml
. Is the build-system
section a requirement that I'm missing...?
It usually should be there if it's describing an installable package, but I'm not sure if it ought to be strictly mandatory here. Looking at PEP 518:
For the vast majority of Python projects that rely upon setuptools, the
pyproject.toml
file will be:
[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"] # PEP 508 specifications.
...
Tools should not require the existence of the
[build-system]
table. Apyproject.toml
file may be used to store configuration details other than build-related data and thus lack a[build-system]
table legitimately. If the file exists but is lacking the[build-system]
table then the default values as specified above should be used. If the table is specified but is missing required fields then the tool should consider it an error.
My thought at the moment is that if you add the above sample section and it works, then this is a bug, since when omitted
. . . the default values as specified above should be used.
A colleague has tracked down the source of the problem. If you check pyproject.toml
with validate-pyproject you get:
[ERROR] project.authors[{data__authors_x}] must be object
Putting the authors
field into the correct format fixes things, i.e.:
authors = [{ name = "author1", email = "author1@email.com" }]
I'm glad you've found the root of the issue. I'll close this since there's no action required regarding pip-tools. Thanks for the issue!
Thanks for your help @atugushev. It may still be worth opening a separate issue re having better error reporting in the case where the pyproject.toml
file is malformed, but obviously it's a less important problem.
$ pip-compile pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /private/var/folders/l0/lnq1ghps5yqc4vkgszlcz92m0000gp/T/tmp.X3gWYzsJ/foo/pyproject.toml
If you think that this could be improved on the build side feel free to report on https://github.com/pypa/build/issues.
Ouch, sorry. This is the pip-tools' message:
pip install .
:
ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
configuration error: `project.authors[{data__authors_x}]` must be object
python -m build
:
ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
configuration error: `project.authors[{data__authors_x}]` must be object
ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist
It may still be worth opening a separate issue re having better error reporting in the case where the
pyproject.toml
file is malformed, but obviously it's a less important problem.
Agreed. Feel free to open a separate issue on the pip-tools tracker.
FYI, there is a tracking issue https://github.com/jazzband/pip-tools/issues/1583
I've stumbled upon this issue a couple of times in the last month. Here is the error I see:
$ pip-compile pyproject.toml -v
Creating venv isolated environment...
Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/OSS/opensearch-logger/pyproject.toml
Environment:
The pyproject.toml
file itself is super simple and passes validation using the validate-pyproject
package:
[build-system]
requires = [
"flit_core >=3.2,<4",
]
build-backend = "flit_core.buildapi"
[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]
It's definitely not a pip-compile
problem as it just calls project_wheel_metadata
from the build
package and then catches the error and reports.
The build
package, in turn, does lots of stuff and ends up calling a Python script called _in_process.py
from the pep517 package.
I have no idea how it is supposed to work, but it looks like the build
package create a one-time temporary virtual environment in which this _in_process.py
script is then invoked. Here is the command with which build
package tries to call a subprocess in my case:
['/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-rj6bf2id/bin/python', '/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpb6wdunjz']
Obviously, when something wrong happens in that subprocess script all we get at the end is:
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/Playground/pip-tools-bug/pyproject.toml
But in reality, the real error does not ever get propagated to the parent process and we don't see it. I've caught it by instrumenting the _in_process.py
file in my virtual environment (lib/python3.11/site-packages/pep517/in_process/_in_process.py
) with print statements.
Exception: description must be specified under [project] or listed as a dynamic field
_in_process.py was called with args: ['/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpttsp4wmf']
Exception: description must be specified under [project] or listed as a dynamic field
File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 338, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/buildapi.py", line 23, in get_requires_for_build_wheel
info = read_flit_config(pyproj_toml)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 79, in read_flit_config
return prep_toml_config(d, path)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 106, in prep_toml_config
loaded_cfg = read_pep621_metadata(d['project'], path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
raise ConfigError(
Now I can see that the underlying error actually comes from the flit
package, complaining about lack of the description
field in the [project]
section of my pyproject.toml
file.
Here is what I manually added to the _in_process.py
file in pep517
package in my venv
def main():
# begin: added
from pathlib import Path
log_file = Path(__file__).parent / "log.txt"
with log_file.open("a") as f:
try:
# end: added
if len(sys.argv) < 3:
sys.exit("Needs args: hook_name, control_dir")
hook_name = sys.argv[1]
control_dir = sys.argv[2]
if hook_name not in HOOK_NAMES:
sys.exit("Unknown hook: %s" % hook_name)
hook = globals()[hook_name]
hook_input = read_json(pjoin(control_dir, 'input.json'))
json_out = {'unsupported': False, 'return_val': None}
try:
json_out['return_val'] = hook(**hook_input['kwargs'])
except BackendUnavailable as e:
json_out['no_backend'] = True
json_out['traceback'] = e.traceback
except BackendInvalid as e:
json_out['backend_invalid'] = True
json_out['backend_error'] = e.message
except GotUnsupportedOperation as e:
json_out['unsupported'] = True
json_out['traceback'] = e.traceback
except HookMissing as e:
json_out['hook_missing'] = True
json_out['missing_hook_name'] = e.hook_name or hook_name
write_json(json_out, pjoin(control_dir, 'output.json'), indent=2)
# begin: more_added
except Exception as e:
f.write(f"Exception: {e}\n")
import traceback
traceback.print_tb(e.__traceback__, file=f)
# end: more_added
I can address the complaints of flit
by adding missing things:
description
fieldopensearch_logger
exists in the same directory where pyproject.toml
is placedAnd voila:
#
# This file is autogenerated by pip-compile with python 3.11
# To update, run:
#
# pip-compile pyproject.toml
#
certifi==2022.9.24
# via
# opensearch-py
# requests
charset-normalizer==2.1.1
# via requests
idna==3.4
# via requests
opensearch-py==2.0.0
# via opensearch-logger (pyproject.toml)
requests==2.28.1
# via opensearch-py
urllib3==1.26.13
# via
# opensearch-py
# requests
Alternatively, should I swap the build system to classic setuptools, it compiles successfully even without said fixes.
[build-system]
requires = [
"setuptools",
"wheel",
]
build-backend = "setuptools.build_meta"
[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]
I don't understand why such system was chosen by creators of build system (not pip-tools). This is the reason we don't see an underlying problem from the chosen build system. It just does not get propagated to higher level tools, such as build
or pip-tools
.
P.S. was so happy to see @atugushev in the comments of this issue. Cheers!
@vduseev thanks for chiming in and for the additional context!
Digging into the issue I found that build.utils.project_wheel_metadata
uses pep517.quiet_subprocess_runner
which suppresses output. With pep517.default_subprocess_runner
It shows the actual error.
Patched build
:
diff --git src/build/util.py src/build/util.py
index 9675393..a435479 100644
--- src/build/util.py
+++ src/build/util.py
@@ -43,7 +43,7 @@ def project_wheel_metadata(
"""
builder = build.ProjectBuilder(
os.fspath(srcdir),
- runner=pep517.quiet_subprocess_runner,
+ runner=pep517.default_subprocess_runner,
)
if not isolated:
pip-compile
output:
root@35878ab01f42:/foo# pip-compile pyproject.toml
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
main()
File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 333, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/usr/local/lib/python3.10/site-packages/flit_core/buildapi.py", line 47, in prepare_metadata_for_build_wheel
ini_info = read_flit_config(pyproj_toml)
File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 79, in read_flit_config
return prep_toml_config(d, path)
File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 106, in prep_toml_config
loaded_cfg = read_pep621_metadata(d['project'], path)
File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
raise ConfigError(
flit_core.config.ConfigError: description must be specified under [project] or listed as a dynamic field
Backend subprocess exited when trying to invoke prepare_metadata_for_build_wheel
Failed to parse /foo/pyproject.toml
@pradyunsg I wonder if would it be okay to introduce one of the following params in build.utils. project_wheel_metadata()
? For example:
project_wheel_metadata(quiet=False)
- do not suppress the outputproject_wheel_metadata(runner=my_runner)
- use my runnerFYI, there is a tracking issue #1583
FTR, while it looks related to #1583 it's affected by a different part of the code. So I've reopened the issue.
FTR: tracking issue https://github.com/pypa/build/issues/553 for project_wheel_metadata
improvement.
@pradyunsg I wonder if would it be okay to introduce one of the following params in
build.utils. project_wheel_metadata()
?
I just saw this, and I would've suggested that you do exactly what you've done. :)
I had a bad whitespace, a '\t' (tab) in a quoted string, that caused this issue; TOML Linter found it.
I found another case today where this error can occur:. I started with a working pyproject.toml
, then I created two subdirectories.
The clue for me was that pip install --editable .
gave an error "multiple top-level packages discovered in a flat-layout".
Running validate-projected returned no errors.
$ cat pyproject.toml
[project]
name = 'test'
description = 'test'
version = '0.1.0'
authors = [ ]
dependencies = [ 'simplejson ' ]
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
# via test (pyproject.toml)
$ mkdir foo
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
# via test (pyproject.toml)
$ mkdir bar
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /home/pbrannan/tmp/test-env/pyproject.toml
$ ./env/bin/python -mvalidate_pyproject pyproject.toml
Valid file: pyproject.toml
Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with
[tool.setuptools]
py-modules = []
Aside from that I'll note that the alternative runner selection is now in build
, but there has not yet been a release including it.
pip-compile omitted all error logs, I have to use pip install -e .
to see what went wrong with the pyproject.toml file. My problem was wrong syntax for fields license
and requires-python
Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with
[tool.setuptools] py-modules = []
Aside from that I'll note that the alternative runner selection is now in
build
, but there has not yet been a release including it.
I also experienced this problem. I have a valid pyproject.toml
with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre's solution above is what worked for me. Thanks @AndydeCleyre!
for the record, build
has implemented switching runner from the default quiet
one, but it's not yet included in any releases of build
.
So the errors come because pip-compile swallows output of pip install -e .
Running the command will reveal the error.
In the case of a flatlayout with multiple packages, mentioned by @cout one can explicitly specify which folders in the flat layout must be included as packages
[tool.setuptools]
packages = ["package1", "package2"]
Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with
[tool.setuptools] py-modules = []
Aside from that I'll note that the alternative runner selection is now in
build
, but there has not yet been a release including it.I also experienced this problem. I have a valid
pyproject.toml
with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre's solution above is what worked for me. Thanks @AndydeCleyre!
this also solved the problem for me.
python -m build
Thanks for that, gives way more descriptive error and it turned out that it my case it was related to write permissions.
I had the same error
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
validate-pyproject
wouldn't yield any issue and the runner=pep517.default_subprocess_runner
trick didn't help.
I rolled back pip-tools to 7.3.0 then for some reason I got a more useful output:
ValueError: Multiple files or folders could be module my_module
Because I had moved my source files from project root to a src folder and after a bit of back and forth in git, the old folder had reappeared empty.
I figured I'd leave a not here in case it helps anyone.
I have been following the guide in the README file on how to use pip-tools with pyproject.toml. Unfortunately, if I add a
project.optional-dependencies
section to pyproject.toml, regardless of what its contents are, I get the following error:Environment Versions
python 3.9.13
pip 22.2.2 from C:\ProgramData\Miniconda3\lib\site-packages\pip (python 3.9)
pip-compile, version 6.9.0
Steps to replicate
project.optional-dependencies
section to yourpyproject.toml
pip-compile pyproject.toml