Closed mtimm closed 2 months ago
jsirois
commented
2 months ago @mtimm I may not need this info, but in case:
mtimm
commented
2 months ago @jsirois
We have a rather distinct docker and package repo setup with our own repos and hardened images. I will need to distill that down to something generic and recreate the issue there. It shouldn't take too long though.
mtimm
commented
2 months ago @jsirois
Here is the distilled Dockerfile:
FROM almalinux:8.10
RUN set -xe && \
dnf install -y epel-release && \
dnf clean all && \
dnf update -y && \
dnf module enable -y ruby:3.1 python27 python36 python38 && \
dnf install -y \
dnf-plugins-core \
gcc \
jq \
pcre2-devel \
systemd \
make \
epel-rpm-macros \
wget \
rpm-build \
rpm-libs \
rpm-sign \
rpm-devel \
rpmdevtools \
libtool \
pcre2-devel \
systemd-devel \
systemd-rpm-macros \
sudo \
which \
ruby \
ruby-devel \
libffi-devel \
python2 \
python2-devel \
python2-pip \
python3-pip \
python36 \
python36-devel \
python38 \
python38-devel \
python3.11 \
python3.11-devel \
python3.11-pip && \
alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 99 && \
alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 100 && \
alternatives --install /usr/bin/python2 python2 /usr/bin/python2.7 100 && \
alternatives --set python3 /usr/bin/python3.11 && \
alternatives --set python /usr/bin/python3 && \
alternatives --set python2 /usr/bin/python2.7 && \
alternatives --add-slave python3 /usr/bin/python3.8 /usr/bin/pip3 pip3 /usr/bin/pip3.8 && \
alternatives --add-slave python3 /usr/bin/python3.11 /usr/bin/pip3 pip3 /usr/bin/pip3.11 && \
pip3 install pip setuptools pex --upgrade && \
gem install --no-document fpm && \
groupadd mock && \
useradd mockbuild -G mock && \
useradd h4ci1 -G wheel && \
echo 'h4ci1 ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && \
echo 'umask u=rwx,g=rx,o=rx' >> /etc/bashrc
RUN pip3.6 install virtualenv
SHELL ["bash", "-c"]
USER h4ci1
WORKDIR /home/h4ci1
HEALTHCHECK CMD uptime
RUN rpmdev-setuptreeI tested it and it had the same issue with v2.19.0 but not with v2.18.1:
[h4ci1@8d1ef3c11b48 ~]$ pex --disable-cache --python=python3.8 --python=python3.11 --platform=manylinux_2_17_x86_64-cp-3.8.13-cp38 --platform=manylinux_2_28_x86_64-cp-3.11.9-cp311 --python-shebang="/usr/bin/env python3" -r pex_requirements.txt -o component_deps.pex
Failed to resolve compatible distributions for 1 target:
1: abbreviated platform cp311-cp311-manylinux_2_28_x86_64 is not compatible with:
cryptography 42.0.8 requires cffi>=1.12; platform_python_implementation != "PyPy" but 4 incompatible dists were resolved:
cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp311-cp311-linux_x86_64.whl
PyNaCl 1.5.0 requires cffi>=1.4.1 but 4 incompatible dists were resolved:
cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp311-cp311-linux_x86_64.whl
paramiko 3.4.0 requires pynacl>=1.5 but 4 incompatible dists were resolved:
PyNaCl-1.5.0-cp311-cp311-linux_x86_64.whl
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
Jinja2 3.1.4 requires MarkupSafe>=2.0 but 4 incompatible dists were resolved:
MarkupSafe-2.1.5-cp311-cp311-linux_x86_64.whl
MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
[h4ci1@8d1ef3c11b48 ~]$ pip install pex==2.18.1 --upgrade
Defaulting to user installation because normal site-packages is not writeable
Collecting pex==2.18.1
Downloading pex-2.18.1-py2.py3-none-any.whl.metadata (7.6 kB)
Downloading pex-2.18.1-py2.py3-none-any.whl (3.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 66.9 MB/s eta 0:00:00
Installing collected packages: pex
Successfully installed pex-2.18.1
[h4ci1@8d1ef3c11b48 ~]$ pex --disable-cache --python=python3.8 --python=python3.11 --platform=manylinux_2_17_x86_64-cp-3.8.13-cp38 --platform=manylinux_2_28_x86_64-cp-3.11.9-cp311 --python-shebang="/usr/bin/env python3" -r pex_requirements.txt -o component_deps.pex
[h4ci1@8d1ef3c11b48 ~]$ echo $?
0
[h4ci1@8d1ef3c11b48 ~]$ ls component_deps.pex
component_deps.pex
[h4ci1@8d1ef3c11b48 ~]$Ok, thanks @mtimm.
In the meantime I reproed with a bit thinner setup:
FROM almalinux:8.10
RUN dnf install -y \
python38-devel \
python3.11-devel \
gcc \
make \
libffi-devel
RUN python3.11 -mvenv /pex/venv && \
/pex/venv/bin/pip install pex
ENV PATH=/pex/venv/bin:$PATH
RUN mkdir /work
WORKDIR /work
COPY pex_requirements.txt .:; docker build -t pex-issues-2532 .
:; docker run --rm -it pex-issues-2532
[root@95e9eb1ce2cc work]# pex -V
2.19.0
[root@95e9eb1ce2cc work]# cat pex_requirements.txt
bcrypt==4.1.3
boto3==1.34.144
botocore==1.34.144
certifi==2024.7.4
cffi==1.16.0
charset-normalizer==3.3.2
configparser==7.0.0
cryptography==42.0.8
decorator==5.1.1
dnspython==2.6.1
docopt==0.6.2
future==1.0.0
futures==3.0.5
hdfs==2.7.3
hvac==0.11.2
idna==3.7
ipaddress==1.0.23
Jinja2==3.1.4
jmespath==1.0.1
MarkupSafe==2.1.5
mock==5.1.0
netaddr==1.3.0
ntplib==0.4.0
paramiko==3.4.0
psutil==6.0.0
py==1.11.0
pycparser==2.22
pycryptodome==3.20.0
PyNaCl==1.5.0
python-consul==1.1.0
python-dateutil==2.9.0.post0
pyvmomi==8.0.3.0.1
PyYAML==6.0.1
requests==2.32.3
retry==0.9.2
s3transfer==0.10.2
setproctitle==1.3.3
six==1.16.0
tornado==6.4
urllib3==1.26.19
packaging==24.1
[root@95e9eb1ce2cc work]# pex --disable-cache --python=python3.8 --python=python3.11 --platform=manylinux_2_17_x86_64-cp-3.8.13-cp38 --platform=manylinux_2_28_x86_64-cp-3.11.9-cp311 --python-shebang="/usr/bin/env python3" -r pex_requirements.txt -o component_deps.pex
Failed to resolve compatible distributions for 1 target:
1: abbreviated platform cp311-cp311-manylinux_2_28_x86_64 is not compatible with:
PyNaCl 1.5.0 requires cffi>=1.4.1 but 4 incompatible dists were resolved:
cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp311-cp311-linux_x86_64.whl
paramiko 3.4.0 requires pynacl>=1.5 but 4 incompatible dists were resolved:
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
PyNaCl-1.5.0-cp311-cp311-linux_x86_64.whl
cryptography 42.0.8 requires cffi>=1.12; platform_python_implementation != "PyPy" but 4 incompatible dists were resolved:
cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi-1.16.0-cp311-cp311-linux_x86_64.whl
Jinja2 3.1.4 requires MarkupSafe>=2.0 but 4 incompatible dists were resolved:
MarkupSafe-2.1.5-cp311-cp311-linux_x86_64.whl
MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
[root@95e9eb1ce2cc work]#I think the new failure is actually correct, but that's only pedantically true. Before drilling in on that, what are you gunning for here at a high level? I ask because there are at least 3 odd things about your pex command:
--platform linux-... --assume-manylinux manylinuxZ, you're using --platform manylinuxZ-.... This is better addressed by 3 below (--platform was a bad idea introduced by Twitter long ago, --complete-platform is what you really want), but guidance here depends on your answer to 1. --complete-platform you're using --platform. You want to use --complete-platform - --platform has always been broken.I'll wait for those answers before going further [^1]. The underlying error though is caused by a stricter post-resolve check introduced in 2.19.0 here: https://github.com/pex-tool/pex/pull/2512/files#diff-25a4139647109115bfa1e1ed672557a1031bc4506ba99e40d4054ae60987ec9aR135. That target.wheel_applies(...) check was not performed previously. The fact your PEX used to work at runtime implies an answer to 1, which is you never needed the pair of --platform ... but I'm not super sure of that conclusion. Especially since you went through the effort of specifying different target manylinux standards for Python 3.8 and 3.11 implying you deploy to machines with different glibc versions installed.
[^1]: I'm happy to continue to hash through this here, but if you want higher bandwidth you can also DM me at https://pex-tool.org/discord.
jsirois
commented
2 months ago One aside is you can also add --ignore-errors to skip this post-resolve check and then you get a PEX built, but I don't recommend that. It just masks what are likely real issues.
jsirois
commented
2 months ago Another way to get this to work with 2.19.0 (or any version) is to add --resolve-local-platforms. This ends up erasing your two --platform options since it finds two corresponding local interpreters that are compatible with the remote foreign platforms, namely --python python3.8 and --python python3.11. This has the added benefit of being faster to build the PEX and making a smaller PEX as well.
As above though, everything hinges on your answer to question 1. Namely what are you actually trying to accomplish with the built PEX. Where should it be able to run?
mtimm
commented
2 months ago Hi @jsirois, I agree our setup may be a bit odd.
Without mudding the waters with too many details, our requirements are that we need this pex file:
Before I go down the complete platform path I wanted to answer your question 1 so you can advise on the best way to accomplish it. Thank you!
jsirois
commented
2 months ago Ok, then if you're building on an Almalinux 8 that already matches one of your two deployment environments, just pex --python python3.11 ... covers that. To also target Centos 7 with the same PEX file, you need to add 1 --platform or --complete-platform that describes that foreign target environment. As strongly hinted, you want to prefer using a --complete-platform. You generate this 1 time (the pex --help mentions this) like so on an examplar Centos target machine:
pex3 interpreter inspect --python python3.8 --tags --markers -i2 > centos7-cp38-glibc2_17.complete-platform.jsonNow, back on the PEX build machine you can say pex --python python3.11 --complete-platform centos7-cp38-glibc2_17.complete-platform.json ... and be assured that you have fully characterized the Centos foreign target platform down to the glibc level. For uniformity, you may want to do the same for the Almalinux target, even though the build machine happens to match today, and generate a complete platform for it: pex3 interpreter inspect --python python3.11 --tags --markers -i2 > almalinux8-cp311-glibc2_28.complete-platform.json. Then you can build your PEX from a Mac even with:
pex \
--complete-platform centos7-cp38-glibc2_17.complete-platform.json \
--complete-platform almalinux8-cp311-glibc2_28.complete-platform.json \
...To give you an idea of the difference between an abbreviated --platform and a --complete-platform:
--platform[root@e46e817dd1c0 work]# python -c '
from pex.platforms import Platform
pl = Platform.create("manylinux_2_28_x86_64-cp-3.11.9-cp311")
print("\n".join(map(str, pl.supported_tags())))
from pex.pep_508 import MarkerEnvironment
print("\n".join(map(str, MarkerEnvironment.from_platform(pl).as_dict().items())))
'
cp311-cp311-manylinux_2_28_x86_64
cp311-abi3-manylinux_2_28_x86_64
cp311-none-manylinux_2_28_x86_64
cp310-abi3-manylinux_2_28_x86_64
cp39-abi3-manylinux_2_28_x86_64
cp38-abi3-manylinux_2_28_x86_64
cp37-abi3-manylinux_2_28_x86_64
cp36-abi3-manylinux_2_28_x86_64
cp35-abi3-manylinux_2_28_x86_64
cp34-abi3-manylinux_2_28_x86_64
cp33-abi3-manylinux_2_28_x86_64
cp32-abi3-manylinux_2_28_x86_64
py311-none-manylinux_2_28_x86_64
py3-none-manylinux_2_28_x86_64
py310-none-manylinux_2_28_x86_64
py39-none-manylinux_2_28_x86_64
py38-none-manylinux_2_28_x86_64
py37-none-manylinux_2_28_x86_64
py36-none-manylinux_2_28_x86_64
py35-none-manylinux_2_28_x86_64
py34-none-manylinux_2_28_x86_64
py33-none-manylinux_2_28_x86_64
py32-none-manylinux_2_28_x86_64
py31-none-manylinux_2_28_x86_64
py30-none-manylinux_2_28_x86_64
cp311-none-any
py311-none-any
py3-none-any
py310-none-any
py39-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any
('implementation_name', 'cpython')
('os_name', 'posix')
('platform_machine', 'x86_64')
('platform_python_implementation', 'CPython')
('platform_system', 'Linux')
('python_full_version', '3.11.9')
('python_version', '3.11')
('sys_platform', 'linux')--complete-platform[root@e46e817dd1c0 work]# pex3 interpreter inspect --python python3.11 --markers --tags -i2
{
"path": "/pex/venv/bin/python3.11",
"compatible_tags": [
"cp311-cp311-manylinux_2_28_x86_64",
"cp311-cp311-manylinux_2_27_x86_64",
"cp311-cp311-manylinux_2_26_x86_64",
"cp311-cp311-manylinux_2_25_x86_64",
"cp311-cp311-manylinux_2_24_x86_64",
"cp311-cp311-manylinux_2_23_x86_64",
"cp311-cp311-manylinux_2_22_x86_64",
"cp311-cp311-manylinux_2_21_x86_64",
"cp311-cp311-manylinux_2_20_x86_64",
"cp311-cp311-manylinux_2_19_x86_64",
"cp311-cp311-manylinux_2_18_x86_64",
"cp311-cp311-manylinux_2_17_x86_64",
"cp311-cp311-manylinux2014_x86_64",
"cp311-cp311-manylinux_2_16_x86_64",
"cp311-cp311-manylinux_2_15_x86_64",
"cp311-cp311-manylinux_2_14_x86_64",
"cp311-cp311-manylinux_2_13_x86_64",
"cp311-cp311-manylinux_2_12_x86_64",
"cp311-cp311-manylinux2010_x86_64",
"cp311-cp311-manylinux_2_11_x86_64",
"cp311-cp311-manylinux_2_10_x86_64",
"cp311-cp311-manylinux_2_9_x86_64",
"cp311-cp311-manylinux_2_8_x86_64",
"cp311-cp311-manylinux_2_7_x86_64",
"cp311-cp311-manylinux_2_6_x86_64",
"cp311-cp311-manylinux_2_5_x86_64",
"cp311-cp311-manylinux1_x86_64",
"cp311-cp311-linux_x86_64",
"cp311-abi3-manylinux_2_28_x86_64",
"cp311-abi3-manylinux_2_27_x86_64",
"cp311-abi3-manylinux_2_26_x86_64",
"cp311-abi3-manylinux_2_25_x86_64",
"cp311-abi3-manylinux_2_24_x86_64",
"cp311-abi3-manylinux_2_23_x86_64",
"cp311-abi3-manylinux_2_22_x86_64",
"cp311-abi3-manylinux_2_21_x86_64",
"cp311-abi3-manylinux_2_20_x86_64",
"cp311-abi3-manylinux_2_19_x86_64",
"cp311-abi3-manylinux_2_18_x86_64",
"cp311-abi3-manylinux_2_17_x86_64",
"cp311-abi3-manylinux2014_x86_64",
"cp311-abi3-manylinux_2_16_x86_64",
"cp311-abi3-manylinux_2_15_x86_64",
"cp311-abi3-manylinux_2_14_x86_64",
"cp311-abi3-manylinux_2_13_x86_64",
"cp311-abi3-manylinux_2_12_x86_64",
"cp311-abi3-manylinux2010_x86_64",
"cp311-abi3-manylinux_2_11_x86_64",
"cp311-abi3-manylinux_2_10_x86_64",
"cp311-abi3-manylinux_2_9_x86_64",
"cp311-abi3-manylinux_2_8_x86_64",
"cp311-abi3-manylinux_2_7_x86_64",
"cp311-abi3-manylinux_2_6_x86_64",
"cp311-abi3-manylinux_2_5_x86_64",
"cp311-abi3-manylinux1_x86_64",
"cp311-abi3-linux_x86_64",
"cp311-none-manylinux_2_28_x86_64",
"cp311-none-manylinux_2_27_x86_64",
"cp311-none-manylinux_2_26_x86_64",
"cp311-none-manylinux_2_25_x86_64",
"cp311-none-manylinux_2_24_x86_64",
"cp311-none-manylinux_2_23_x86_64",
"cp311-none-manylinux_2_22_x86_64",
"cp311-none-manylinux_2_21_x86_64",
"cp311-none-manylinux_2_20_x86_64",
"cp311-none-manylinux_2_19_x86_64",
"cp311-none-manylinux_2_18_x86_64",
"cp311-none-manylinux_2_17_x86_64",
"cp311-none-manylinux2014_x86_64",
"cp311-none-manylinux_2_16_x86_64",
"cp311-none-manylinux_2_15_x86_64",
"cp311-none-manylinux_2_14_x86_64",
"cp311-none-manylinux_2_13_x86_64",
"cp311-none-manylinux_2_12_x86_64",
"cp311-none-manylinux2010_x86_64",
"cp311-none-manylinux_2_11_x86_64",
"cp311-none-manylinux_2_10_x86_64",
"cp311-none-manylinux_2_9_x86_64",
"cp311-none-manylinux_2_8_x86_64",
"cp311-none-manylinux_2_7_x86_64",
"cp311-none-manylinux_2_6_x86_64",
"cp311-none-manylinux_2_5_x86_64",
"cp311-none-manylinux1_x86_64",
"cp311-none-linux_x86_64",
"cp310-abi3-manylinux_2_28_x86_64",
"cp310-abi3-manylinux_2_27_x86_64",
"cp310-abi3-manylinux_2_26_x86_64",
"cp310-abi3-manylinux_2_25_x86_64",
"cp310-abi3-manylinux_2_24_x86_64",
"cp310-abi3-manylinux_2_23_x86_64",
"cp310-abi3-manylinux_2_22_x86_64",
"cp310-abi3-manylinux_2_21_x86_64",
"cp310-abi3-manylinux_2_20_x86_64",
"cp310-abi3-manylinux_2_19_x86_64",
"cp310-abi3-manylinux_2_18_x86_64",
"cp310-abi3-manylinux_2_17_x86_64",
"cp310-abi3-manylinux2014_x86_64",
"cp310-abi3-manylinux_2_16_x86_64",
"cp310-abi3-manylinux_2_15_x86_64",
"cp310-abi3-manylinux_2_14_x86_64",
"cp310-abi3-manylinux_2_13_x86_64",
"cp310-abi3-manylinux_2_12_x86_64",
"cp310-abi3-manylinux2010_x86_64",
"cp310-abi3-manylinux_2_11_x86_64",
"cp310-abi3-manylinux_2_10_x86_64",
"cp310-abi3-manylinux_2_9_x86_64",
"cp310-abi3-manylinux_2_8_x86_64",
"cp310-abi3-manylinux_2_7_x86_64",
"cp310-abi3-manylinux_2_6_x86_64",
"cp310-abi3-manylinux_2_5_x86_64",
"cp310-abi3-manylinux1_x86_64",
"cp310-abi3-linux_x86_64",
"cp39-abi3-manylinux_2_28_x86_64",
"cp39-abi3-manylinux_2_27_x86_64",
"cp39-abi3-manylinux_2_26_x86_64",
"cp39-abi3-manylinux_2_25_x86_64",
"cp39-abi3-manylinux_2_24_x86_64",
"cp39-abi3-manylinux_2_23_x86_64",
"cp39-abi3-manylinux_2_22_x86_64",
"cp39-abi3-manylinux_2_21_x86_64",
"cp39-abi3-manylinux_2_20_x86_64",
"cp39-abi3-manylinux_2_19_x86_64",
"cp39-abi3-manylinux_2_18_x86_64",
"cp39-abi3-manylinux_2_17_x86_64",
"cp39-abi3-manylinux2014_x86_64",
"cp39-abi3-manylinux_2_16_x86_64",
"cp39-abi3-manylinux_2_15_x86_64",
"cp39-abi3-manylinux_2_14_x86_64",
"cp39-abi3-manylinux_2_13_x86_64",
"cp39-abi3-manylinux_2_12_x86_64",
"cp39-abi3-manylinux2010_x86_64",
"cp39-abi3-manylinux_2_11_x86_64",
"cp39-abi3-manylinux_2_10_x86_64",
"cp39-abi3-manylinux_2_9_x86_64",
"cp39-abi3-manylinux_2_8_x86_64",
"cp39-abi3-manylinux_2_7_x86_64",
"cp39-abi3-manylinux_2_6_x86_64",
"cp39-abi3-manylinux_2_5_x86_64",
"cp39-abi3-manylinux1_x86_64",
"cp39-abi3-linux_x86_64",
"cp38-abi3-manylinux_2_28_x86_64",
"cp38-abi3-manylinux_2_27_x86_64",
"cp38-abi3-manylinux_2_26_x86_64",
"cp38-abi3-manylinux_2_25_x86_64",
"cp38-abi3-manylinux_2_24_x86_64",
"cp38-abi3-manylinux_2_23_x86_64",
"cp38-abi3-manylinux_2_22_x86_64",
"cp38-abi3-manylinux_2_21_x86_64",
"cp38-abi3-manylinux_2_20_x86_64",
"cp38-abi3-manylinux_2_19_x86_64",
"cp38-abi3-manylinux_2_18_x86_64",
"cp38-abi3-manylinux_2_17_x86_64",
"cp38-abi3-manylinux2014_x86_64",
"cp38-abi3-manylinux_2_16_x86_64",
"cp38-abi3-manylinux_2_15_x86_64",
"cp38-abi3-manylinux_2_14_x86_64",
"cp38-abi3-manylinux_2_13_x86_64",
"cp38-abi3-manylinux_2_12_x86_64",
"cp38-abi3-manylinux2010_x86_64",
"cp38-abi3-manylinux_2_11_x86_64",
"cp38-abi3-manylinux_2_10_x86_64",
"cp38-abi3-manylinux_2_9_x86_64",
"cp38-abi3-manylinux_2_8_x86_64",
"cp38-abi3-manylinux_2_7_x86_64",
"cp38-abi3-manylinux_2_6_x86_64",
"cp38-abi3-manylinux_2_5_x86_64",
"cp38-abi3-manylinux1_x86_64",
"cp38-abi3-linux_x86_64",
"cp37-abi3-manylinux_2_28_x86_64",
"cp37-abi3-manylinux_2_27_x86_64",
"cp37-abi3-manylinux_2_26_x86_64",
"cp37-abi3-manylinux_2_25_x86_64",
"cp37-abi3-manylinux_2_24_x86_64",
"cp37-abi3-manylinux_2_23_x86_64",
"cp37-abi3-manylinux_2_22_x86_64",
"cp37-abi3-manylinux_2_21_x86_64",
"cp37-abi3-manylinux_2_20_x86_64",
"cp37-abi3-manylinux_2_19_x86_64",
"cp37-abi3-manylinux_2_18_x86_64",
"cp37-abi3-manylinux_2_17_x86_64",
"cp37-abi3-manylinux2014_x86_64",
"cp37-abi3-manylinux_2_16_x86_64",
"cp37-abi3-manylinux_2_15_x86_64",
"cp37-abi3-manylinux_2_14_x86_64",
"cp37-abi3-manylinux_2_13_x86_64",
"cp37-abi3-manylinux_2_12_x86_64",
"cp37-abi3-manylinux2010_x86_64",
"cp37-abi3-manylinux_2_11_x86_64",
"cp37-abi3-manylinux_2_10_x86_64",
"cp37-abi3-manylinux_2_9_x86_64",
"cp37-abi3-manylinux_2_8_x86_64",
"cp37-abi3-manylinux_2_7_x86_64",
"cp37-abi3-manylinux_2_6_x86_64",
"cp37-abi3-manylinux_2_5_x86_64",
"cp37-abi3-manylinux1_x86_64",
"cp37-abi3-linux_x86_64",
"cp36-abi3-manylinux_2_28_x86_64",
"cp36-abi3-manylinux_2_27_x86_64",
"cp36-abi3-manylinux_2_26_x86_64",
"cp36-abi3-manylinux_2_25_x86_64",
"cp36-abi3-manylinux_2_24_x86_64",
"cp36-abi3-manylinux_2_23_x86_64",
"cp36-abi3-manylinux_2_22_x86_64",
"cp36-abi3-manylinux_2_21_x86_64",
"cp36-abi3-manylinux_2_20_x86_64",
"cp36-abi3-manylinux_2_19_x86_64",
"cp36-abi3-manylinux_2_18_x86_64",
"cp36-abi3-manylinux_2_17_x86_64",
"cp36-abi3-manylinux2014_x86_64",
"cp36-abi3-manylinux_2_16_x86_64",
"cp36-abi3-manylinux_2_15_x86_64",
"cp36-abi3-manylinux_2_14_x86_64",
"cp36-abi3-manylinux_2_13_x86_64",
"cp36-abi3-manylinux_2_12_x86_64",
"cp36-abi3-manylinux2010_x86_64",
"cp36-abi3-manylinux_2_11_x86_64",
"cp36-abi3-manylinux_2_10_x86_64",
"cp36-abi3-manylinux_2_9_x86_64",
"cp36-abi3-manylinux_2_8_x86_64",
"cp36-abi3-manylinux_2_7_x86_64",
"cp36-abi3-manylinux_2_6_x86_64",
"cp36-abi3-manylinux_2_5_x86_64",
"cp36-abi3-manylinux1_x86_64",
"cp36-abi3-linux_x86_64",
"cp35-abi3-manylinux_2_28_x86_64",
"cp35-abi3-manylinux_2_27_x86_64",
"cp35-abi3-manylinux_2_26_x86_64",
"cp35-abi3-manylinux_2_25_x86_64",
"cp35-abi3-manylinux_2_24_x86_64",
"cp35-abi3-manylinux_2_23_x86_64",
"cp35-abi3-manylinux_2_22_x86_64",
"cp35-abi3-manylinux_2_21_x86_64",
"cp35-abi3-manylinux_2_20_x86_64",
"cp35-abi3-manylinux_2_19_x86_64",
"cp35-abi3-manylinux_2_18_x86_64",
"cp35-abi3-manylinux_2_17_x86_64",
"cp35-abi3-manylinux2014_x86_64",
"cp35-abi3-manylinux_2_16_x86_64",
"cp35-abi3-manylinux_2_15_x86_64",
"cp35-abi3-manylinux_2_14_x86_64",
"cp35-abi3-manylinux_2_13_x86_64",
"cp35-abi3-manylinux_2_12_x86_64",
"cp35-abi3-manylinux2010_x86_64",
"cp35-abi3-manylinux_2_11_x86_64",
"cp35-abi3-manylinux_2_10_x86_64",
"cp35-abi3-manylinux_2_9_x86_64",
"cp35-abi3-manylinux_2_8_x86_64",
"cp35-abi3-manylinux_2_7_x86_64",
"cp35-abi3-manylinux_2_6_x86_64",
"cp35-abi3-manylinux_2_5_x86_64",
"cp35-abi3-manylinux1_x86_64",
"cp35-abi3-linux_x86_64",
"cp34-abi3-manylinux_2_28_x86_64",
"cp34-abi3-manylinux_2_27_x86_64",
"cp34-abi3-manylinux_2_26_x86_64",
"cp34-abi3-manylinux_2_25_x86_64",
"cp34-abi3-manylinux_2_24_x86_64",
"cp34-abi3-manylinux_2_23_x86_64",
"cp34-abi3-manylinux_2_22_x86_64",
"cp34-abi3-manylinux_2_21_x86_64",
"cp34-abi3-manylinux_2_20_x86_64",
"cp34-abi3-manylinux_2_19_x86_64",
"cp34-abi3-manylinux_2_18_x86_64",
"cp34-abi3-manylinux_2_17_x86_64",
"cp34-abi3-manylinux2014_x86_64",
"cp34-abi3-manylinux_2_16_x86_64",
"cp34-abi3-manylinux_2_15_x86_64",
"cp34-abi3-manylinux_2_14_x86_64",
"cp34-abi3-manylinux_2_13_x86_64",
"cp34-abi3-manylinux_2_12_x86_64",
"cp34-abi3-manylinux2010_x86_64",
"cp34-abi3-manylinux_2_11_x86_64",
"cp34-abi3-manylinux_2_10_x86_64",
"cp34-abi3-manylinux_2_9_x86_64",
"cp34-abi3-manylinux_2_8_x86_64",
"cp34-abi3-manylinux_2_7_x86_64",
"cp34-abi3-manylinux_2_6_x86_64",
"cp34-abi3-manylinux_2_5_x86_64",
"cp34-abi3-manylinux1_x86_64",
"cp34-abi3-linux_x86_64",
"cp33-abi3-manylinux_2_28_x86_64",
"cp33-abi3-manylinux_2_27_x86_64",
"cp33-abi3-manylinux_2_26_x86_64",
"cp33-abi3-manylinux_2_25_x86_64",
"cp33-abi3-manylinux_2_24_x86_64",
"cp33-abi3-manylinux_2_23_x86_64",
"cp33-abi3-manylinux_2_22_x86_64",
"cp33-abi3-manylinux_2_21_x86_64",
"cp33-abi3-manylinux_2_20_x86_64",
"cp33-abi3-manylinux_2_19_x86_64",
"cp33-abi3-manylinux_2_18_x86_64",
"cp33-abi3-manylinux_2_17_x86_64",
"cp33-abi3-manylinux2014_x86_64",
"cp33-abi3-manylinux_2_16_x86_64",
"cp33-abi3-manylinux_2_15_x86_64",
"cp33-abi3-manylinux_2_14_x86_64",
"cp33-abi3-manylinux_2_13_x86_64",
"cp33-abi3-manylinux_2_12_x86_64",
"cp33-abi3-manylinux2010_x86_64",
"cp33-abi3-manylinux_2_11_x86_64",
"cp33-abi3-manylinux_2_10_x86_64",
"cp33-abi3-manylinux_2_9_x86_64",
"cp33-abi3-manylinux_2_8_x86_64",
"cp33-abi3-manylinux_2_7_x86_64",
"cp33-abi3-manylinux_2_6_x86_64",
"cp33-abi3-manylinux_2_5_x86_64",
"cp33-abi3-manylinux1_x86_64",
"cp33-abi3-linux_x86_64",
"cp32-abi3-manylinux_2_28_x86_64",
"cp32-abi3-manylinux_2_27_x86_64",
"cp32-abi3-manylinux_2_26_x86_64",
"cp32-abi3-manylinux_2_25_x86_64",
"cp32-abi3-manylinux_2_24_x86_64",
"cp32-abi3-manylinux_2_23_x86_64",
"cp32-abi3-manylinux_2_22_x86_64",
"cp32-abi3-manylinux_2_21_x86_64",
"cp32-abi3-manylinux_2_20_x86_64",
"cp32-abi3-manylinux_2_19_x86_64",
"cp32-abi3-manylinux_2_18_x86_64",
"cp32-abi3-manylinux_2_17_x86_64",
"cp32-abi3-manylinux2014_x86_64",
"cp32-abi3-manylinux_2_16_x86_64",
"cp32-abi3-manylinux_2_15_x86_64",
"cp32-abi3-manylinux_2_14_x86_64",
"cp32-abi3-manylinux_2_13_x86_64",
"cp32-abi3-manylinux_2_12_x86_64",
"cp32-abi3-manylinux2010_x86_64",
"cp32-abi3-manylinux_2_11_x86_64",
"cp32-abi3-manylinux_2_10_x86_64",
"cp32-abi3-manylinux_2_9_x86_64",
"cp32-abi3-manylinux_2_8_x86_64",
"cp32-abi3-manylinux_2_7_x86_64",
"cp32-abi3-manylinux_2_6_x86_64",
"cp32-abi3-manylinux_2_5_x86_64",
"cp32-abi3-manylinux1_x86_64",
"cp32-abi3-linux_x86_64",
"py311-none-manylinux_2_28_x86_64",
"py311-none-manylinux_2_27_x86_64",
"py311-none-manylinux_2_26_x86_64",
"py311-none-manylinux_2_25_x86_64",
"py311-none-manylinux_2_24_x86_64",
"py311-none-manylinux_2_23_x86_64",
"py311-none-manylinux_2_22_x86_64",
"py311-none-manylinux_2_21_x86_64",
"py311-none-manylinux_2_20_x86_64",
"py311-none-manylinux_2_19_x86_64",
"py311-none-manylinux_2_18_x86_64",
"py311-none-manylinux_2_17_x86_64",
"py311-none-manylinux2014_x86_64",
"py311-none-manylinux_2_16_x86_64",
"py311-none-manylinux_2_15_x86_64",
"py311-none-manylinux_2_14_x86_64",
"py311-none-manylinux_2_13_x86_64",
"py311-none-manylinux_2_12_x86_64",
"py311-none-manylinux2010_x86_64",
"py311-none-manylinux_2_11_x86_64",
"py311-none-manylinux_2_10_x86_64",
"py311-none-manylinux_2_9_x86_64",
"py311-none-manylinux_2_8_x86_64",
"py311-none-manylinux_2_7_x86_64",
"py311-none-manylinux_2_6_x86_64",
"py311-none-manylinux_2_5_x86_64",
"py311-none-manylinux1_x86_64",
"py311-none-linux_x86_64",
"py3-none-manylinux_2_28_x86_64",
"py3-none-manylinux_2_27_x86_64",
"py3-none-manylinux_2_26_x86_64",
"py3-none-manylinux_2_25_x86_64",
"py3-none-manylinux_2_24_x86_64",
"py3-none-manylinux_2_23_x86_64",
"py3-none-manylinux_2_22_x86_64",
"py3-none-manylinux_2_21_x86_64",
"py3-none-manylinux_2_20_x86_64",
"py3-none-manylinux_2_19_x86_64",
"py3-none-manylinux_2_18_x86_64",
"py3-none-manylinux_2_17_x86_64",
"py3-none-manylinux2014_x86_64",
"py3-none-manylinux_2_16_x86_64",
"py3-none-manylinux_2_15_x86_64",
"py3-none-manylinux_2_14_x86_64",
"py3-none-manylinux_2_13_x86_64",
"py3-none-manylinux_2_12_x86_64",
"py3-none-manylinux2010_x86_64",
"py3-none-manylinux_2_11_x86_64",
"py3-none-manylinux_2_10_x86_64",
"py3-none-manylinux_2_9_x86_64",
"py3-none-manylinux_2_8_x86_64",
"py3-none-manylinux_2_7_x86_64",
"py3-none-manylinux_2_6_x86_64",
"py3-none-manylinux_2_5_x86_64",
"py3-none-manylinux1_x86_64",
"py3-none-linux_x86_64",
"py310-none-manylinux_2_28_x86_64",
"py310-none-manylinux_2_27_x86_64",
"py310-none-manylinux_2_26_x86_64",
"py310-none-manylinux_2_25_x86_64",
"py310-none-manylinux_2_24_x86_64",
"py310-none-manylinux_2_23_x86_64",
"py310-none-manylinux_2_22_x86_64",
"py310-none-manylinux_2_21_x86_64",
"py310-none-manylinux_2_20_x86_64",
"py310-none-manylinux_2_19_x86_64",
"py310-none-manylinux_2_18_x86_64",
"py310-none-manylinux_2_17_x86_64",
"py310-none-manylinux2014_x86_64",
"py310-none-manylinux_2_16_x86_64",
"py310-none-manylinux_2_15_x86_64",
"py310-none-manylinux_2_14_x86_64",
"py310-none-manylinux_2_13_x86_64",
"py310-none-manylinux_2_12_x86_64",
"py310-none-manylinux2010_x86_64",
"py310-none-manylinux_2_11_x86_64",
"py310-none-manylinux_2_10_x86_64",
"py310-none-manylinux_2_9_x86_64",
"py310-none-manylinux_2_8_x86_64",
"py310-none-manylinux_2_7_x86_64",
"py310-none-manylinux_2_6_x86_64",
"py310-none-manylinux_2_5_x86_64",
"py310-none-manylinux1_x86_64",
"py310-none-linux_x86_64",
"py39-none-manylinux_2_28_x86_64",
"py39-none-manylinux_2_27_x86_64",
"py39-none-manylinux_2_26_x86_64",
"py39-none-manylinux_2_25_x86_64",
"py39-none-manylinux_2_24_x86_64",
"py39-none-manylinux_2_23_x86_64",
"py39-none-manylinux_2_22_x86_64",
"py39-none-manylinux_2_21_x86_64",
"py39-none-manylinux_2_20_x86_64",
"py39-none-manylinux_2_19_x86_64",
"py39-none-manylinux_2_18_x86_64",
"py39-none-manylinux_2_17_x86_64",
"py39-none-manylinux2014_x86_64",
"py39-none-manylinux_2_16_x86_64",
"py39-none-manylinux_2_15_x86_64",
"py39-none-manylinux_2_14_x86_64",
"py39-none-manylinux_2_13_x86_64",
"py39-none-manylinux_2_12_x86_64",
"py39-none-manylinux2010_x86_64",
"py39-none-manylinux_2_11_x86_64",
"py39-none-manylinux_2_10_x86_64",
"py39-none-manylinux_2_9_x86_64",
"py39-none-manylinux_2_8_x86_64",
"py39-none-manylinux_2_7_x86_64",
"py39-none-manylinux_2_6_x86_64",
"py39-none-manylinux_2_5_x86_64",
"py39-none-manylinux1_x86_64",
"py39-none-linux_x86_64",
"py38-none-manylinux_2_28_x86_64",
"py38-none-manylinux_2_27_x86_64",
"py38-none-manylinux_2_26_x86_64",
"py38-none-manylinux_2_25_x86_64",
"py38-none-manylinux_2_24_x86_64",
"py38-none-manylinux_2_23_x86_64",
"py38-none-manylinux_2_22_x86_64",
"py38-none-manylinux_2_21_x86_64",
"py38-none-manylinux_2_20_x86_64",
"py38-none-manylinux_2_19_x86_64",
"py38-none-manylinux_2_18_x86_64",
"py38-none-manylinux_2_17_x86_64",
"py38-none-manylinux2014_x86_64",
"py38-none-manylinux_2_16_x86_64",
"py38-none-manylinux_2_15_x86_64",
"py38-none-manylinux_2_14_x86_64",
"py38-none-manylinux_2_13_x86_64",
"py38-none-manylinux_2_12_x86_64",
"py38-none-manylinux2010_x86_64",
"py38-none-manylinux_2_11_x86_64",
"py38-none-manylinux_2_10_x86_64",
"py38-none-manylinux_2_9_x86_64",
"py38-none-manylinux_2_8_x86_64",
"py38-none-manylinux_2_7_x86_64",
"py38-none-manylinux_2_6_x86_64",
"py38-none-manylinux_2_5_x86_64",
"py38-none-manylinux1_x86_64",
"py38-none-linux_x86_64",
"py37-none-manylinux_2_28_x86_64",
"py37-none-manylinux_2_27_x86_64",
"py37-none-manylinux_2_26_x86_64",
"py37-none-manylinux_2_25_x86_64",
"py37-none-manylinux_2_24_x86_64",
"py37-none-manylinux_2_23_x86_64",
"py37-none-manylinux_2_22_x86_64",
"py37-none-manylinux_2_21_x86_64",
"py37-none-manylinux_2_20_x86_64",
"py37-none-manylinux_2_19_x86_64",
"py37-none-manylinux_2_18_x86_64",
"py37-none-manylinux_2_17_x86_64",
"py37-none-manylinux2014_x86_64",
"py37-none-manylinux_2_16_x86_64",
"py37-none-manylinux_2_15_x86_64",
"py37-none-manylinux_2_14_x86_64",
"py37-none-manylinux_2_13_x86_64",
"py37-none-manylinux_2_12_x86_64",
"py37-none-manylinux2010_x86_64",
"py37-none-manylinux_2_11_x86_64",
"py37-none-manylinux_2_10_x86_64",
"py37-none-manylinux_2_9_x86_64",
"py37-none-manylinux_2_8_x86_64",
"py37-none-manylinux_2_7_x86_64",
"py37-none-manylinux_2_6_x86_64",
"py37-none-manylinux_2_5_x86_64",
"py37-none-manylinux1_x86_64",
"py37-none-linux_x86_64",
"py36-none-manylinux_2_28_x86_64",
"py36-none-manylinux_2_27_x86_64",
"py36-none-manylinux_2_26_x86_64",
"py36-none-manylinux_2_25_x86_64",
"py36-none-manylinux_2_24_x86_64",
"py36-none-manylinux_2_23_x86_64",
"py36-none-manylinux_2_22_x86_64",
"py36-none-manylinux_2_21_x86_64",
"py36-none-manylinux_2_20_x86_64",
"py36-none-manylinux_2_19_x86_64",
"py36-none-manylinux_2_18_x86_64",
"py36-none-manylinux_2_17_x86_64",
"py36-none-manylinux2014_x86_64",
"py36-none-manylinux_2_16_x86_64",
"py36-none-manylinux_2_15_x86_64",
"py36-none-manylinux_2_14_x86_64",
"py36-none-manylinux_2_13_x86_64",
"py36-none-manylinux_2_12_x86_64",
"py36-none-manylinux2010_x86_64",
"py36-none-manylinux_2_11_x86_64",
"py36-none-manylinux_2_10_x86_64",
"py36-none-manylinux_2_9_x86_64",
"py36-none-manylinux_2_8_x86_64",
"py36-none-manylinux_2_7_x86_64",
"py36-none-manylinux_2_6_x86_64",
"py36-none-manylinux_2_5_x86_64",
"py36-none-manylinux1_x86_64",
"py36-none-linux_x86_64",
"py35-none-manylinux_2_28_x86_64",
"py35-none-manylinux_2_27_x86_64",
"py35-none-manylinux_2_26_x86_64",
"py35-none-manylinux_2_25_x86_64",
"py35-none-manylinux_2_24_x86_64",
"py35-none-manylinux_2_23_x86_64",
"py35-none-manylinux_2_22_x86_64",
"py35-none-manylinux_2_21_x86_64",
"py35-none-manylinux_2_20_x86_64",
"py35-none-manylinux_2_19_x86_64",
"py35-none-manylinux_2_18_x86_64",
"py35-none-manylinux_2_17_x86_64",
"py35-none-manylinux2014_x86_64",
"py35-none-manylinux_2_16_x86_64",
"py35-none-manylinux_2_15_x86_64",
"py35-none-manylinux_2_14_x86_64",
"py35-none-manylinux_2_13_x86_64",
"py35-none-manylinux_2_12_x86_64",
"py35-none-manylinux2010_x86_64",
"py35-none-manylinux_2_11_x86_64",
"py35-none-manylinux_2_10_x86_64",
"py35-none-manylinux_2_9_x86_64",
"py35-none-manylinux_2_8_x86_64",
"py35-none-manylinux_2_7_x86_64",
"py35-none-manylinux_2_6_x86_64",
"py35-none-manylinux_2_5_x86_64",
"py35-none-manylinux1_x86_64",
"py35-none-linux_x86_64",
"py34-none-manylinux_2_28_x86_64",
"py34-none-manylinux_2_27_x86_64",
"py34-none-manylinux_2_26_x86_64",
"py34-none-manylinux_2_25_x86_64",
"py34-none-manylinux_2_24_x86_64",
"py34-none-manylinux_2_23_x86_64",
"py34-none-manylinux_2_22_x86_64",
"py34-none-manylinux_2_21_x86_64",
"py34-none-manylinux_2_20_x86_64",
"py34-none-manylinux_2_19_x86_64",
"py34-none-manylinux_2_18_x86_64",
"py34-none-manylinux_2_17_x86_64",
"py34-none-manylinux2014_x86_64",
"py34-none-manylinux_2_16_x86_64",
"py34-none-manylinux_2_15_x86_64",
"py34-none-manylinux_2_14_x86_64",
"py34-none-manylinux_2_13_x86_64",
"py34-none-manylinux_2_12_x86_64",
"py34-none-manylinux2010_x86_64",
"py34-none-manylinux_2_11_x86_64",
"py34-none-manylinux_2_10_x86_64",
"py34-none-manylinux_2_9_x86_64",
"py34-none-manylinux_2_8_x86_64",
"py34-none-manylinux_2_7_x86_64",
"py34-none-manylinux_2_6_x86_64",
"py34-none-manylinux_2_5_x86_64",
"py34-none-manylinux1_x86_64",
"py34-none-linux_x86_64",
"py33-none-manylinux_2_28_x86_64",
"py33-none-manylinux_2_27_x86_64",
"py33-none-manylinux_2_26_x86_64",
"py33-none-manylinux_2_25_x86_64",
"py33-none-manylinux_2_24_x86_64",
"py33-none-manylinux_2_23_x86_64",
"py33-none-manylinux_2_22_x86_64",
"py33-none-manylinux_2_21_x86_64",
"py33-none-manylinux_2_20_x86_64",
"py33-none-manylinux_2_19_x86_64",
"py33-none-manylinux_2_18_x86_64",
"py33-none-manylinux_2_17_x86_64",
"py33-none-manylinux2014_x86_64",
"py33-none-manylinux_2_16_x86_64",
"py33-none-manylinux_2_15_x86_64",
"py33-none-manylinux_2_14_x86_64",
"py33-none-manylinux_2_13_x86_64",
"py33-none-manylinux_2_12_x86_64",
"py33-none-manylinux2010_x86_64",
"py33-none-manylinux_2_11_x86_64",
"py33-none-manylinux_2_10_x86_64",
"py33-none-manylinux_2_9_x86_64",
"py33-none-manylinux_2_8_x86_64",
"py33-none-manylinux_2_7_x86_64",
"py33-none-manylinux_2_6_x86_64",
"py33-none-manylinux_2_5_x86_64",
"py33-none-manylinux1_x86_64",
"py33-none-linux_x86_64",
"py32-none-manylinux_2_28_x86_64",
"py32-none-manylinux_2_27_x86_64",
"py32-none-manylinux_2_26_x86_64",
"py32-none-manylinux_2_25_x86_64",
"py32-none-manylinux_2_24_x86_64",
"py32-none-manylinux_2_23_x86_64",
"py32-none-manylinux_2_22_x86_64",
"py32-none-manylinux_2_21_x86_64",
"py32-none-manylinux_2_20_x86_64",
"py32-none-manylinux_2_19_x86_64",
"py32-none-manylinux_2_18_x86_64",
"py32-none-manylinux_2_17_x86_64",
"py32-none-manylinux2014_x86_64",
"py32-none-manylinux_2_16_x86_64",
"py32-none-manylinux_2_15_x86_64",
"py32-none-manylinux_2_14_x86_64",
"py32-none-manylinux_2_13_x86_64",
"py32-none-manylinux_2_12_x86_64",
"py32-none-manylinux2010_x86_64",
"py32-none-manylinux_2_11_x86_64",
"py32-none-manylinux_2_10_x86_64",
"py32-none-manylinux_2_9_x86_64",
"py32-none-manylinux_2_8_x86_64",
"py32-none-manylinux_2_7_x86_64",
"py32-none-manylinux_2_6_x86_64",
"py32-none-manylinux_2_5_x86_64",
"py32-none-manylinux1_x86_64",
"py32-none-linux_x86_64",
"py31-none-manylinux_2_28_x86_64",
"py31-none-manylinux_2_27_x86_64",
"py31-none-manylinux_2_26_x86_64",
"py31-none-manylinux_2_25_x86_64",
"py31-none-manylinux_2_24_x86_64",
"py31-none-manylinux_2_23_x86_64",
"py31-none-manylinux_2_22_x86_64",
"py31-none-manylinux_2_21_x86_64",
"py31-none-manylinux_2_20_x86_64",
"py31-none-manylinux_2_19_x86_64",
"py31-none-manylinux_2_18_x86_64",
"py31-none-manylinux_2_17_x86_64",
"py31-none-manylinux2014_x86_64",
"py31-none-manylinux_2_16_x86_64",
"py31-none-manylinux_2_15_x86_64",
"py31-none-manylinux_2_14_x86_64",
"py31-none-manylinux_2_13_x86_64",
"py31-none-manylinux_2_12_x86_64",
"py31-none-manylinux2010_x86_64",
"py31-none-manylinux_2_11_x86_64",
"py31-none-manylinux_2_10_x86_64",
"py31-none-manylinux_2_9_x86_64",
"py31-none-manylinux_2_8_x86_64",
"py31-none-manylinux_2_7_x86_64",
"py31-none-manylinux_2_6_x86_64",
"py31-none-manylinux_2_5_x86_64",
"py31-none-manylinux1_x86_64",
"py31-none-linux_x86_64",
"py30-none-manylinux_2_28_x86_64",
"py30-none-manylinux_2_27_x86_64",
"py30-none-manylinux_2_26_x86_64",
"py30-none-manylinux_2_25_x86_64",
"py30-none-manylinux_2_24_x86_64",
"py30-none-manylinux_2_23_x86_64",
"py30-none-manylinux_2_22_x86_64",
"py30-none-manylinux_2_21_x86_64",
"py30-none-manylinux_2_20_x86_64",
"py30-none-manylinux_2_19_x86_64",
"py30-none-manylinux_2_18_x86_64",
"py30-none-manylinux_2_17_x86_64",
"py30-none-manylinux2014_x86_64",
"py30-none-manylinux_2_16_x86_64",
"py30-none-manylinux_2_15_x86_64",
"py30-none-manylinux_2_14_x86_64",
"py30-none-manylinux_2_13_x86_64",
"py30-none-manylinux_2_12_x86_64",
"py30-none-manylinux2010_x86_64",
"py30-none-manylinux_2_11_x86_64",
"py30-none-manylinux_2_10_x86_64",
"py30-none-manylinux_2_9_x86_64",
"py30-none-manylinux_2_8_x86_64",
"py30-none-manylinux_2_7_x86_64",
"py30-none-manylinux_2_6_x86_64",
"py30-none-manylinux_2_5_x86_64",
"py30-none-manylinux1_x86_64",
"py30-none-linux_x86_64",
"cp311-none-any",
"py311-none-any",
"py3-none-any",
"py310-none-any",
"py39-none-any",
"py38-none-any",
"py37-none-any",
"py36-none-any",
"py35-none-any",
"py34-none-any",
"py33-none-any",
"py32-none-any",
"py31-none-any",
"py30-none-any"
],
"marker_environment": {
"implementation_name": "cpython",
"implementation_version": "3.11.9",
"os_name": "posix",
"platform_machine": "x86_64",
"platform_python_implementation": "CPython",
"platform_release": "5.15.153.1-microsoft-standard-WSL2",
"platform_system": "Linux",
"platform_version": "#1 SMP Fri Mar 29 23:14:13 UTC 2024",
"python_full_version": "3.11.9",
"python_version": "3.11",
"sys_platform": "linux"
}
}It's only the latter that contains all the information needed to properly resolve and check distributions. The former has big holes!
mtimm
commented
2 months ago @jsirois thank you very much. These extra details are super helpful. I think the one point I did not fully think through was that pex3 needs to be run on the target machine to generate the complete platform details, but now that I reflect on it, that totally makes sense.
I will give this a try in a bit and close the issue after verifying it works like we want.
jsirois
commented
2 months ago So, @mtimm to explain why this worked before with the abbreviated --platform manylinux_2_28_x86_64-cp-3.11.9-cp311 but does not now.
Previously, the PEX build would proceed as follows:
--platform manylinux_2_28_x86_64-cp-3.11.9-cp311 case, Pex would fail to find a pre-built manylinux_2_28_x86_64 (exact match) wheel on PyPI, and proceed to build from source.--python python3.11 case, Pex would succeed in resolving a manylinux_2_17 pre-built wheel from PyPIWith 2.19.0, Pex eagerly checks the resolved manylinux_2_17 foreign platform wheel matches the abbreviated --platform manylinux_2_28_x86_64-cp-3.11.9-cp311 and fails. You can see why it fails from the --platform details I showed above. For the abbreviated platform, the only known tags are the exact manylinux_2_28_x86_64 tags, and not the descending backwards compatible set that can only be calculated on the foreign target [^1].
[^1]: The way tags are calculated for --platform targets is by running pip -v debug ..., like so:
:; pip -v debug --platform manylinux_2_28_x86_64 --python-version 3.11 --abi cp311 --implementation cp
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 24.2 from /home/jsirois/support/pex/issues/2532/pip.venv/lib/python3.11/site-packages/pip (python 3.11)
sys.version: 3.11.10 (main, Sep 13 2024, 18:39:29) [GCC 13.2.0]
sys.executable: /home/jsirois/support/pex/issues/2532/pip.venv/bin/python
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /home/jsirois/support/pex/issues/2532/pip.venv/lib/python3.11/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
CacheControl==0.14.0
distlib==0.3.8
distro==1.9.0
msgpack==1.0.8
packaging==24.1
platformdirs==4.2.2
pyproject-hooks==1.0.0
requests==2.32.3
certifi==2024.07.04
idna==3.7
urllib3==1.26.18
rich==13.7.1 (Unable to locate actual module version, using vendor.txt specified version)
pygments==2.18.0
typing_extensions==4.12.2 (Unable to locate actual module version, using vendor.txt specified version)
resolvelib==1.0.1
setuptools==70.3.0 (Unable to locate actual module version, using vendor.txt specified version)
tomli==2.0.1
truststore==0.9.1
Compatible tags: 39 (target: platforms=['manylinux_2_28_x86_64'] version_info='3.11' abis=['cp311'] implementation='cp')
cp311-cp311-manylinux_2_28_x86_64
cp311-abi3-manylinux_2_28_x86_64
cp311-none-manylinux_2_28_x86_64
cp310-abi3-manylinux_2_28_x86_64
cp39-abi3-manylinux_2_28_x86_64
cp38-abi3-manylinux_2_28_x86_64
cp37-abi3-manylinux_2_28_x86_64
cp36-abi3-manylinux_2_28_x86_64
cp35-abi3-manylinux_2_28_x86_64
cp34-abi3-manylinux_2_28_x86_64
cp33-abi3-manylinux_2_28_x86_64
cp32-abi3-manylinux_2_28_x86_64
py311-none-manylinux_2_28_x86_64
py3-none-manylinux_2_28_x86_64
py310-none-manylinux_2_28_x86_64
py39-none-manylinux_2_28_x86_64
py38-none-manylinux_2_28_x86_64
py37-none-manylinux_2_28_x86_64
py36-none-manylinux_2_28_x86_64
py35-none-manylinux_2_28_x86_64
py34-none-manylinux_2_28_x86_64
py33-none-manylinux_2_28_x86_64
py32-none-manylinux_2_28_x86_64
py31-none-manylinux_2_28_x86_64
py30-none-manylinux_2_28_x86_64
cp311-none-any
py311-none-any
py3-none-any
py310-none-any
py39-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-anyOk, well - hopefully this all makes more sense now.
That said, Pex lives by the rule of not breaking people as it has done here!
I think it may make sense to skip the new target.wheel_applies(...) check added in 2.19.0 for abbreviated --platform since these are generally broken and there really isn't enough information to know for sure. Maybe checking, but just logging a warning when the --platform target.wheel_applies(...) check fails. I could use that warning to both not break folks, but also nudge them to use --complete-platform instead. @mtimm what do you think about that?
mtimm
commented
2 months ago I like the idea of a warning. This issue would likely give enough info for others who are struggling like I was to come up with a workable solution for this sort of requirement.
Thanks again.
jsirois
commented
2 months ago Alrighty, #2533 fixes the regression by adding a warning. I should have a release out in a few hours. Thanks for the report @mtimm.
mtimm
commented
2 months ago Thanks for the quick response @jsirois and taking the time to help fill in my knowledge gaps. I have tested the changes I need to make and it all seems to work as expected with 2.19.0 using complete-platform. Feel free to close this issue when you have the release out. Thanks again!
jsirois
commented
2 months ago Ok, the fix for --platform is out in 2.19.1: https://github.com/pex-tool/pex/releases/tag/v2.19.1
We build a pex file using an almalinux 8.10 container with python 3.8 and python 3.11 installed with the command:
pex -v --disable-cache --python=python3.8 --python=python3.11 --platform=manylinux_2_17_x86_64-cp-3.8.13-cp38 --platform=manylinux_2_28_x86_64-cp-3.11.9-cp311 --python-shebang="/usr/bin/env python3" -r pex_requirements.txt -o component_deps.pexThis was functional up until pex v2.19.0.
With version pex v2.19.0 we now get:
I think in previous versions these dependencies were resolved by building them at the time the pex file is being built. It looks like they are being built by v2.19.0 based on debug output if I increase the verbosity. It seems like wheels built as part of the pex build are not being found to resolve the dependencies once they are built. At least that is my best guess.
The pex_requirements.txt we are using is: