Closed Github-dm-CDE closed 2 months ago
Please try adding --pip-version latest
to your Pex command line. I can explain what's going on later.
Hi @jsirois, thank you for the quick feedback and the suggested solution!
In fact, adding the --pip-version latest
argument fixed the problem. 😊
We were also wondering whether there is a way to use the pip version already available in the venv.
We are very much looking forward to your explanation 😉
But we cannot explain why it is called during the pex build process.
It's called during any Python build or resolve process because methodtools 0.4.7 is a source distribution: https://pypi.org/project/methodtools/0.4.7/#files. In order for Pip to resolve methodtools dependencies or Pex to build a wheel from it (Pex requires wheels), the sdist must be ~built. Building the sdist requires evaluating its setup.py.
So that's the why. The reason this is a problem in Pex (with default settings) and not Pip is Pex's default settings are backwards compatible to Pex 2.0.0 which used a patched version of Pip 20.3.4, and - more to the point, a patched version of setuptools. That patched version of setuptools is here: https://github.com/pex-tool/pex/tree/main/pex/vendor/_vendored/setuptools/setuptools-44.0.0+3acb925dd708430aeaf197ea53ac8a752f7c1863.dist-info and the version is 44.0.0+3acb925dd708430aeaf197ea53ac8a752f7c1863
. The methodtools setup.py
code assume a version of the form X.Y.Z
with X, Y and Z all being integers. Here, the Pex vendored patched version of setuptools has a +3ac...
local version identifier suffix that defeats the methodtools setup.py
assumption.
The fix --pip-version latest
could in fact be --pip-version <any version except the default>
, see pex --help
for all the supported values of --pip-version
. All versions of Pip except the default use a production release of setuptools with a version in the standard X.Y.Z
form; thus the fix.
You probably want to read more of the Pex CLI --help
since features and options are added frequently to improve performance and compatibility, but they are never defaulted to since Pex never wants to break existing users.
@Github-dm-CDE I'm going to close this as an answered question, but please speak up if this is still confusing.
Hi @jsirois , thank you very much for your explanation!
Since the update to
apache-airflow
2.9.1methodtools
0.4.7 is required as a transitive dependency. Currently there seems to be a problem in the interaction between the mentioned methodtools version andpex
2.3.1 which we cannot quite explain. Our pex build fails every time since the update.This is our
requirements.txt
:This is our build script: (We are currently using pex version 2.3.1)
And this is the resulting error message:
The problem seems to result from this section of the methodtools package: https://github.com/youknowone/methodtools/blob/0.4.7/setup.py But we cannot explain why it is called during the pex build process.
Does anyone have a clue as to why this could be or can they give us a hint?