1.6.0: Error installing `develop` packages from pyproject.toml

mpuhacz commented 1 year ago

poetry version: 1.6.1
poetry-plugin-export version: 1.6.0
python version: 3.11.6

Issue

After updating poetry-plugin-export to version 1.6.0, we began encountering errors when attempting to pip install packages generated with the export command. This happens specifically for packages added to pyproject.toml with develop = true attribute. The export command creates a requirements.txt entry with pinned commit hash, while pip install tries to install from the main branch as specified in pyproject.toml.

Could we adjust the behavior for packages with develop = true in pyproject.toml to utilize the reference (e.g., "main") over the resolved_reference (specific commit hash) for their dependecies when creating the requirements.txt?

Error

Pip install command is failing with the following error:

116.2 INFO: pip is looking at multiple versions of customer-kafka to determine which version is compatible with other requirements. This could take a while.
120.4 ERROR: Cannot install -r /tmp/requirements.txt (line 31) and glue-schema-registry 1.1.7 (from git+ssh://****@github.com/our-repo-com/glue-schema-registry.git@a675fed50b70718a7f568a01fa92fd4e485e54a4) because these package versions have conflicting dependencies.
120.4
120.4 The conflict is caused by:
120.4     The user requested  1.1.7 (from git+ssh://****@github.com/our-repo-com/glue-schema-registry.git@a675fed50b70718a7f568a01fa92fd4e485e54a4)
120.4     customer-kafka 0.1.0 depends on glue-schema-registry 1.1.7 (from git+ssh://****@github.com/our-repo-com/glue-schema-registry.git@main)
120.4
120.4 To fix this you could try to:
120.4 1. loosen the range of package versions you've specified
120.4 2. remove package versions to allow pip attempt to solve the dependency conflict

Configuration

The package is added in develop mode in pyproject.toml.

customer-kafka = {path = "../../shared/customer-kafka", develop = true}

The glue-schema-registry dependecy is defined inside the customer-kafka, with the following entry in pyproject.toml.

[tool.poetry.dependencies]
glue-schema-registry = {git = "git@github.com:our-repo-com/glue-schema-registry.git", rev = "main"}

and the poetry.lock:

[[package]]
name = "glue-schema-registry"
version = "1.1.7"
description = ""
optional = false
python-versions = ">=3.7"
files = []
develop = false

[package.dependencies]
boto3 = ">=1.17.102"
fastavro = ">=1.4.5"
fastjsonschema = ">=2.15,<3.0"
orjson = ">=3.8.12"

[package.extras]
dev = ["flake8 (>=3)", "pytest (>=6)"]
kafka-python = ["kafka-python (>=2)"]

[package.source]
type = "git"
url = "git@github.com:our-repo-com/glue-schema-registry.git"
reference = "main"
resolved_reference = "a675fed50b70718a7f568a01fa92fd4e485e54a4"

radoering commented 1 year ago

Does it work if you run pip install with the --no-deps flag?

I can't yet follow why this is specific to develop packages. Can you please provide the generated requirements.txt?

mpuhacz commented 1 year ago

Unfortunately it fails too:

 => ERROR [build 12/12] RUN --mount=type=ssh pip install -r /tmp/requirements.txt --no-deps --target /build                                                                                                             37.8s
------
 > [build 12/12] RUN --mount=type=ssh pip install -r /tmp/requirements.txt --no-deps --target /build:
2.096 Collecting glue-schema-registry@ git+ssh://git@github.com/our-repo-com/glue-schema-registry.git@a675fed50b70718a7f568a01fa92fd4e485e54a4 (from -r /tmp/requirements.txt (line 3))
2.098   Cloning ssh://****@github.com/our-repo-com/glue-schema-registry.git (to revision a675fed50b70718a7f568a01fa92fd4e485e54a4) to /tmp/pip-install-aqenhbnl/glue-schema-registry_1da94e0b5d13475fbb9c047f3ea032fc
2.126   Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/our-repo-com/glue-schema-registry.git' /tmp/pip-install-aqenhbnl/glue-schema-registry_1da94e0b5d13475fbb9c047f3ea032fc
2.698   Warning: Permanently added the RSA host key for IP address '140.82.121.4' to the list of known hosts.
6.126   Running command git rev-parse -q --verify 'sha^a675fed50b70718a7f568a01fa92fd4e485e54a4'
6.164   Running command git fetch -q 'ssh://****@github.com/our-repo-com/glue-schema-registry.git' a675fed50b70718a7f568a01fa92fd4e485e54a4
8.002   Resolved ssh://****@github.com/our-repo-com/glue-schema-registry.git to commit a675fed50b70718a7f568a01fa92fd4e485e54a4
8.048   Installing build dependencies: started
16.55   Installing build dependencies: finished with status 'done'
16.56   Getting requirements to build wheel: started
17.29   Getting requirements to build wheel: finished with status 'done'
17.32   Preparing metadata (pyproject.toml): started
18.20   Preparing metadata (pyproject.toml): finished with status 'done'
18.21 Ignoring colorama: markers 'python_version >= "3.11" and python_version < "4.0" and (sys_platform == "win32" or platform_system == "Windows")' don't match your environment
18.24 Processing /app/ledger (from -r /tmp/requirements.txt (line 29))
18.25   Installing build dependencies: started
23.33   Installing build dependencies: finished with status 'done'
23.33   Getting requirements to build wheel: started
23.86   Getting requirements to build wheel: finished with status 'done'
23.86   Preparing metadata (pyproject.toml): started
24.99   Preparing metadata (pyproject.toml): finished with status 'done'
25.02 Processing /shared/customer-dynamodb (from -r /tmp/requirements.txt (line 30))
25.02   Installing build dependencies: started
29.72   Installing build dependencies: finished with status 'done'
29.73   Getting requirements to build wheel: started
30.26   Getting requirements to build wheel: finished with status 'done'
30.27   Preparing metadata (pyproject.toml): started
31.35   Preparing metadata (pyproject.toml): finished with status 'done'
31.38 Processing /shared/customer-kafka (from -r /tmp/requirements.txt (line 31))
31.39   Installing build dependencies: started
36.18   Installing build dependencies: finished with status 'done'
36.19   Getting requirements to build wheel: started
36.73   Getting requirements to build wheel: finished with status 'done'
36.73   Preparing metadata (pyproject.toml): started
37.48   Preparing metadata (pyproject.toml): finished with status 'error'
37.51   error: subprocess-exited-with-error
37.51
37.51   × Preparing metadata (pyproject.toml) did not run successfully.
37.51   │ exit code: 1
37.51   ╰─> [17 lines of output]
37.51       Traceback (most recent call last):
37.51         File "/var/lang/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
37.51           main()
37.51         File "/var/lang/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
37.51           json_out['return_val'] = hook(**hook_input['kwargs'])
37.51                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
37.51         File "/var/lang/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
37.51           return hook(metadata_directory, config_settings)
37.51                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
37.51         File "/tmp/pip-build-env-7ycggcvw/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 42, in prepare_metadata_for_build_wheel
37.51           poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
37.51                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
37.51         File "/tmp/pip-build-env-7ycggcvw/overlay/lib/python3.11/site-packages/poetry/core/factory.py", line 58, in create_poetry
37.51           raise RuntimeError("The Poetry configuration is invalid:\n" + message)
37.51       RuntimeError: The Poetry configuration is invalid:
37.51         - data.dependencies.glue-schema-registry must be valid exactly by one definition (0 matches found)
37.51
37.51       [end of output]
37.51
37.51   note: This error originates from a subprocess, and is likely not a problem with pip.
37.52 error: metadata-generation-failed
37.52
37.52 × Encountered error while generating package metadata.
37.52 ╰─> See above for output.
37.52
37.52 note: This is an issue with the package mentioned above, not pip.
37.52 hint: See above for details.
37.56

poetry export --format requirements.txt --without-hashes --output /tmp/requirements.txt

anyio==4.0.0 ; python_version >= "3.11" and python_version < "4.0"
aws-lambda-typing==2.18.0 ; python_version >= "3.11" and python_version < "4.0"
glue-schema-registry @ git+ssh://git@github.com/our-repo-com/glue-schema-registry.git@a675fed50b70718a7f568a01fa92fd4e485e54a4 ; python_version >= "3.11" and python_version < "4.0"
boto3-stubs[dynamodb]==1.28.63 ; python_version >= "3.11" and python_version < "4.0"
boto3==1.28.63 ; python_version >= "3.11" and python_version < "4.0"
botocore-stubs==1.31.63 ; python_version >= "3.11" and python_version < "4.0"
botocore==1.31.63 ; python_version >= "3.11" and python_version < "4.0"
cachetools==5.3.1 ; python_version >= "3.11" and python_version < "4.0"
certifi==2023.7.22 ; python_version >= "3.11" and python_version < "4.0"
charset-normalizer==3.3.0 ; python_version >= "3.11" and python_version < "4.0"
click==8.1.7 ; python_version >= "3.11" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.11" and python_version < "4.0" and (sys_platform == "win32" or platform_system == "Windows")
confluent-kafka==2.2.0 ; python_version >= "3.11" and python_version < "4.0"
dnspython==2.4.2 ; python_version >= "3.11" and python_version < "4.0"
email-validator==2.0.0.post2 ; python_version >= "3.11" and python_version < "4.0"
fastapi[all]==0.95.2 ; python_version >= "3.11" and python_version < "4.0"
fastavro==1.8.4 ; python_version >= "3.11" and python_version < "4.0"
fastjsonschema==2.18.1 ; python_version >= "3.11" and python_version < "4.0"
h11==0.14.0 ; python_version >= "3.11" and python_version < "4.0"
httpcore==0.18.0 ; python_version >= "3.11" and python_version < "4.0"
httptools==0.6.0 ; python_version >= "3.11" and python_version < "4.0"
httpx==0.25.0 ; python_version >= "3.11" and python_version < "4.0"
idna==3.4 ; python_version >= "3.11" and python_version < "4.0"
iso4217==1.11.20220401 ; python_version >= "3.11" and python_version < "4.0"
itsdangerous==2.1.2 ; python_version >= "3.11" and python_version < "4.0"
jinja2==3.1.2 ; python_version >= "3.11" and python_version < "4.0"
jmespath==1.0.1 ; python_version >= "3.11" and python_version < "4.0"
kafka-python==2.0.2 ; python_version >= "3.11" and python_version < "4.0"
ledger @ file:///app/ledger ; python_version >= "3.11" and python_version < "4.0"
customer-dynamodb @ file:///shared/customer-dynamodb ; python_version >= "3.11" and python_version < "4.0"
customer-kafka @ file:///shared/customer-kafka ; python_version >= "3.11" and python_version < "4.0"
mangum==0.16.0 ; python_version >= "3.11" and python_version < "4.0"
markupsafe==2.1.3 ; python_version >= "3.11" and python_version < "4.0"
mypy-boto3-dynamodb==1.28.55 ; python_version >= "3.11" and python_version < "4.0"
orjson==3.9.9 ; python_version >= "3.11" and python_version < "4.0"
pendulum==2.1.2 ; python_version >= "3.11" and python_version < "4.0"
pydantic==1.10.13 ; python_version >= "3.11" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.11" and python_version < "4.0"
python-decouple==3.8 ; python_version >= "3.11" and python_version < "4.0"
python-dotenv==1.0.0 ; python_version >= "3.11" and python_version < "4.0"
python-multipart==0.0.6 ; python_version >= "3.11" and python_version < "4.0"
pytzdata==2020.1 ; python_version >= "3.11" and python_version < "4.0"
pyyaml==6.0.1 ; python_version >= "3.11" and python_version < "4.0"
requests==2.31.0 ; python_version >= "3.11" and python_version < "4.0"
s3transfer==0.7.0 ; python_version >= "3.11" and python_version < "4.0"
sentry-sdk==1.32.0 ; python_version >= "3.11" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.11" and python_version < "4.0"
sniffio==1.3.0 ; python_version >= "3.11" and python_version < "4.0"
starlette==0.27.0 ; python_version >= "3.11" and python_version < "4.0"
structlog==23.2.0 ; python_version >= "3.11" and python_version < "4.0"
tenacity==8.2.3 ; python_version >= "3.11" and python_version < "4.0"
types-awscrt==0.19.2 ; python_version >= "3.11" and python_version < "4.0"
types-s3transfer==0.7.0 ; python_version >= "3.11" and python_version < "4.0"
typing-extensions==4.5.0 ; python_version >= "3.11" and python_version < "4.0"
ujson==5.8.0 ; python_version >= "3.11" and python_version < "4.0"
urllib3==2.0.7 ; python_version >= "3.11" and python_version < "4.0"
uvicorn[standard]==0.20.0 ; python_version >= "3.11" and python_version < "4.0"
uvloop==0.18.0 ; (sys_platform != "win32" and sys_platform != "cygwin") and platform_python_implementation != "PyPy" and python_version >= "3.11" and python_version < "4.0"
watchfiles==0.21.0 ; python_version >= "3.11" and python_version < "4.0"
websockets==11.0.3 ; python_version >= "3.11" and python_version < "4.0"

The 1.5.0 version would return:

glue-schema-registry @ git+ssh://git@github.com/our-repo-com/glue-schema-registry.git@main ; python_version >= "3.11" and python_version < "4.0"

dimbleby commented 1 year ago

data.dependencies.glue-schema-registry must be valid exactly by one definition (0 matches found)

this looks completely different from what you first reported. Here it seems as though the new json validation is stricter than the old, I think it is unhappy becuase the git property does not conform to the uri format as the schema requires.

I suppose the schema was always wrong to require this. Or maybe the fastjsonschema validation is wrong, I'm not sure.

Edit: the first of these I reckon. There's no scheme at the front of this string, it's not a URI.

mpuhacz commented 1 year ago

@dimbleby that was it! Thank you! Do you know why this error showed up just today? Is it safe to assume this is related to https://github.com/python-poetry/poetry-core/releases/tag/1.8.0 relase?

So for anyone looking for a resolution - If you are using poetry export to later do pip install make sure to pass --no-deps flag to the pip install command.

radoering commented 1 year ago

Do you know why this error showed up just today? Is it safe to assume this is related to https://github.com/python-poetry/poetry-core/releases/tag/1.8.0 relase?

Yes, it is. We switched from jsonschema to fastjsonschema, which seems to do stricter validation. This should be fixed now with https://github.com/python-poetry/poetry-core/releases/tag/1.8.1

python-poetry / poetry-plugin-export