Github-dm-CDE
commented
1 month ago The current workaround for us is to use the older v1.7.1 version of poetry-plugin-export with poetry self add poetry-plugin-export==1.7.1.
Open Github-dm-CDE opened 1 month ago
Github-dm-CDE
commented
1 month ago The current workaround for us is to use the older v1.7.1 version of poetry-plugin-export with poetry self add poetry-plugin-export==1.7.1.
jsirois
commented
1 month ago @Github-dm-CDE can you please provide the exact Pex command line you use to build the PEX? Normally, instead of doing a poetry export then Pex build using the exported requirements, you'd just say pex /path/to/poetry/project ... in one step and skip the export altogether.
jsirois
commented
1 month ago @Github-dm-CDE I just looked at Pex's requirements.txt parsing code and and it would seem to handle -e ... correctly: https://github.com/pex-tool/pex/blob/36dd2374569b5f3fbef54918e6ba3bd1ff852b61/pex/requirements.py#L264-L269. Now Pants, on the other hand, does not: https://github.com/pantsbuild/pants/blob/bef63e96042d6f79975a662172b4cd85e8ecd0a3/src/python/pants/util/requirements.py#L8-L17
@Github-dm-CDE are you building the PEX using Pants? If so, that's where the bug is. If you can confirm you're using Pants I'll close this as an answered question and you'll have to raise an issue with the Pants project; otherwise, please provide the Pex command line you use and I'll repro and debug.
Github-dm-CDE
commented
1 month ago @Github-dm-CDE can you please provide the exact Pex command line you use to build the PEX? Normally, instead of doing a poetry export then Pex build using the exported requirements, you'd just say
pex /path/to/poetry/project ...in one step and skip the export altogether.
We are building our pex with help of this small bash script:
#!/bin/sh
set -e
rm -rf test_dist && mkdir test_dist
pip install -U pip
poetry install --only build
poetry export --with dev --extras sources -f requirements.txt --output test_dist/test_requirements.txt --without-hashes --with-credentials
echo "========= BUILD PEX ========="
poetry run pex -r test_dist/test_requirements.txt -o test_dist/test_sources_dependencies.pex
This is the content of our pyproject.toml
[tool.poetry]
name = "spark-sources"
version = "0"
description = "spark-sources for CXA-CK Cluster jobs"
readme = "README.md"
authors = ["<[Masked]@[Masked]>"]
[tool.poetry.dependencies]
python = "~3.9"
bda-base-dependencies = { path = "bda_base_dependencies/", develop = true, optional = true }
ck-spark-sources = { path = "sources/", develop = true, optional = true }
utils = { path = "utils/", develop = true, optional = true }
common-utils = { path = "utils/common_utils/", develop = true, optional = true }
datahub-utils = { path = "utils/datahub_utils/", develop = true, optional = true }
kafka-utils = { path = "utils/kafka_utils/", develop = true, optional = true }
logck = { path = "utils/logck/", develop = true, optional = true }
mssql-utils = { path = "utils/mssql_utils/", develop = true, optional = true }
pyspark = { version = "3.3.0", optional = true }
table-builder = { path = "table_builder/", develop = true, optional = true }
test-helpers = { path = "utils/test_helpers/", develop = true, optional = true }
zeppelin-utils = { path = "utils/zeppelin_utils/", develop = true, optional = true }
[tool.poetry.extras]
bda-base = ["bda-base-dependencies"]
pyspark = ["pyspark"]
table-builder = ["bda-base-dependencies", "common-utils", "datahub-utils", "pyspark", "table-builder", "zeppelin-utils"]
sources = [
"bda-base-dependencies",
"ck-spark-sources",
"pyspark",
"table-builder",
"utils",
]
[tool.poetry.group.build.dependencies]
pex = "^2.1.117"
[tool.poetry.group.dev.dependencies]
Flask = "^3.0.0"
freezegun = "^1.1.0"
pytest = "^8.0.0"
pytest_check = "^2.0.0"
pytest-cov = "^5.0.0"
pre-commit = "^3.3.3"
rich = "^13.4.0"
ipdb = "^0.13.9"
dm-livy-submit = { version = "^2.3.0", source = "ck-pypi" }
dm-knox-token = { version = "^2.0.1", source = "ck-pypi" }
pytest-flaskserver = "^1.1.3"
sphinx = "^7.1.2"
autodoc-pydantic = "^2.0.1"
sphinx-rtd-theme = "^2.0.0"
sphinx-autoapi = "^3.0.0"
myst-parser = "^3.0.0"
[tool.pytest.ini_options]
addopts = "-ra -v --showlocals -s"
markers = [
"local_test_only", # marks tests that should not run as system test
"system_test_only" # marks tests that should only run as system tests on cluster
]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[[tool.poetry.source]]
name = "dm-artifactory-pypi"
url = "https://[Masked].com/artifactory/api/pypi/pypi-remote/simple"
priority = "supplemental"
[[tool.poetry.source]]
name = "ck-pypi"
url = "https://[Masked]/api/v4/projects/13599/packages/pypi/simple/"
priority = "primary"@Github-dm-CDE where is the test_helpers module / package found? Is that loose on disk or is it supposed to be in one of the file:/// projects packaged in the Pex?
The more information you can provide the better. For example, PEX's are just zips. You might create both the good PEX and bad PEX, unzip them both in different directories and then run a recursive diff to see what's different. That would be very helpful information.
xebab
commented
1 day ago Hi @jsirois ,
I have been picking up this issue and I have a minimal example for you. I have checked out an arbitrary Python project (requests in this case) into my local /tmp/ directory and then I have built a PEX with the command pex -r requirements.txt -o foo.pex using the requirements.txt below
pytest
-e file:///tmp/requestsNow pex-tools foo.pex info shows that the requests library and its dependencies are included in the PEX.
{
"bootstrap_hash": "abae6f731d4b0d818ab6d02e09294593143e0132",
"build_properties": {
"pex_version": "2.9.0"
},
"code_hash": "4155f382ffd2ab38bcf15a249b859595d933456e",
"deps_are_wheel_files": false,
"distributions": {
"certifi-2024.7.4-py3-none-any.whl": "f51955747a040ad60cc9b281d2b81748fda55a5bb5c4b6b10be65f0354609302",
"charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "606334df13ed33c5ab6385635909cf91fb213ef7b11022f40e7a3161c7b2ee98",
"exceptiongroup-1.2.1-py3-none-any.whl": "f74bb5f1b9d4d177eb069806e20fd043b788e4275f0bd3405badabfc0103dac3",
"idna-3.7-py3-none-any.whl": "243156850dfb4ab00c4b9604d58d83e722e41eb0f445358feb23af5701231307",
"iniconfig-2.0.0-py3-none-any.whl": "988e40b3b55723f6802760cae3cd0319699d3e69ae7f34c7793d38fb44d579b7",
"packaging-24.1-py3-none-any.whl": "25baba288a0f0e2ebe1b0117cad380c5d4c0137aff4a2c7755f92cffd0eb38f8",
"pluggy-1.5.0-py3-none-any.whl": "6826f70172987927d47205f5bc5b1b18c380201149e5b63bc90e4d1645e7a8d9",
"pytest-8.2.2-py3-none-any.whl": "ce97a2c0ec29fd42f042f7e76e7f857fb83919d9054169b16988d628f47b43f4",
"requests-2.32.1-py3-none-any.whl": "7226b06e0a6f282f84ca65816c130920e5d01c2932e0382e9fe7afec1c993928",
"tomli-2.0.1-py3-none-any.whl": "8af30168c90e438c0488ddb0dcc8bfbdb7f54299570473943266c8f800de58dc",
"urllib3-2.2.2-py3-none-any.whl": "568f866565573953519a3419f2ab438633b3fb6e932725d0a1aecb46a35d9b12"
},
"emit_warnings": true,
"excluded": [],
"ignore_errors": false,
"includes_tools": false,
"inherit_path": "false",
"inject_args": [],
"inject_env": {},
"inject_python_args": [],
"interpreter_constraints": [],
"max_install_jobs": 1,
"overridden": [],
"pex_hash": "3f284033281410a36e3ac7a6efebbc97277d6311",
"pex_path": "",
"pex_paths": [],
"requirements": [
"pytest"
],
"strip_pex_env": true,
"venv": false,
"venv_bin_path": "false",
"venv_copies": false,
"venv_hermetic_scripts": true,
"venv_site_packages_copies": false,
"pex_root": "/home/alex/.pex"
}However there is an error importing the package, because it is not part of the requirements array.
> ./foo.pex
Python 3.9.16 (main, Oct 10 2023, 14:24:18)
[GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
Traceback (most recent call last):
File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'Hope that helps to debug the issue.
jsirois
commented
1 day ago Thanks @xebab, that's a helpful repro case. When I get back to my keyboard on 7/14 I'll take a look and hopefully get a quick fix out.
Since https://github.com/python-poetry/poetry-plugin-export/pull/258 was merged, the temporary requirements.txt format's changed. This makes local project dependencies unparseable. The build works, but the pex entrypoints do not exist / changed.
requirements.txt with poetry-plugin-export@1.7.1
requirements.txt with poetry-plugin-export@1.8.0
With the pex created based on the requierements.txt of v1.8.0, our pipeline failed: