Pipenv install of Python 3.9 fails using ubuntu AMI

[david-mcclure]

Issue description

We have a solution deployed on AWS where an EC2 instance uses docker files to build images for different components. One of those components has a docker script we haven't changed in over a year that, among other things, installs python 3.9 using pipenv. We ran it as recently as two days ago without any problems. When we tried running today, we got an error we've never seen before from the pipenv install.

Any help would be greatly appreciated. Thanks!

Expected result

When script runs "RUN pipenv install --python 3.9" that it succeeds. See log from last successful run:

15:02:54 #15 [11/12] RUN pipenv install --python 3.9 15:02:55 #15 0.914 Warning: the environment variable LANG is not set! 15:02:55 #15 0.914 We recommend setting this in ~/.profile (or equivalent) for proper expected behavior. 15:02:55 #15 1.290 Creating a virtualenv for this project... 15:02:55 #15 1.290 Pipfile: /code/Pipfile 15:02:55 #15 1.316 Using /usr/bin/python3.9 (3.9.5) to create virtualenv... 15:02:56 #15 2.101 created virtual environment CPython3.9.5.final.0-64 in 476ms 15:02:56 #15 2.101 creator CPython3Posix(dest=/root/.local/share/virtualenvs/code-_Py8Si6I, clear=False, no_vcs_ignore=False, global=False) 15:02:56 #15 2.101 seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv) 15:02:56 #15 2.101 added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0 15:02:56 #15 2.101 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator 15:02:56 #15 2.101 15:02:56 #15 2.102 ✔ Successfully created virtual environment! 15:02:56 #15 2.157 Virtualenv location: /root/.local/share/virtualenvs/code-_Py8Si6I 15:02:56 #15 2.208 To activate this project's virtualenv, run pipenv shell. 15:02:56 #15 2.208 Alternatively, run a command inside the virtualenv with pipenv run. 15:02:56 #15 2.220 To activate this project's virtualenv, run pipenv shell. 15:02:56 #15 2.220 Alternatively, run a command inside the virtualenv with pipenv run. 15:02:56 #15 2.223 Installing dependencies from Pipfile.lock (d65a77)...

Actual result

When script runs "RUN pipenv install --python 3.9", it fails. See log from last run:

15:49:43 #15 [11/12] RUN pipenv install --python 3.9 15:49:44 #15 0.935 Warning: the environment variable LANG is not set! 15:49:44 #15 0.935 We recommend setting this in ~/.profile (or equivalent) for proper expected behavior. 15:49:45 #15 1.345 Creating a virtualenv for this project... 15:49:45 #15 1.346 Pipfile: /code/Pipfile 15:49:45 #15 1.387 Using /usr/bin/python3.9 (3.9.5) to create virtualenv... 15:49:45 #15 1.767 15:49:45 #15 1.767 ✘ Failed creating virtual environment 15:49:45 #15 1.767 [pipenv.exceptions.VirtualenvCreationException]: Traceback (most recent call last): 15:49:45 #15 1.767 File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main 15:49:45 #15 1.767 mod_name, mod_spec, code = _get_module_details(mod_name, _Error) 15:49:45 #15 1.767 File "/usr/lib/python3.9/runpy.py", line 147, in _get_module_details 15:49:45 #15 1.767 return _get_module_details(pkg_main_name, error) 15:49:45 #15 1.767 File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details 15:49:45 #15 1.767 import(pkg_name) 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/init.py", line 3, in 15:49:45 #15 1.767 from .run import cli_run, session_via_cli 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/run/init.py", line 15, in 15:49:45 #15 1.767 from .plugin.creators import CreatorSelector 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/run/plugin/creators.py", line 7, in 15:49:45 #15 1.767 from virtualenv.create.via_global_ref.builtin.builtin_way import VirtualenvBuiltin 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/create/via_global_ref/builtin/builtin_way.py", line 5, in 15:49:45 #15 1.767 from virtualenv.create.creator import Creator 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/create/creator.py", line 13, in 15:49:45 #15 1.767 from virtualenv.discovery.cached_py_info import LogCmd 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/discovery/cached_py_info.py", line 25, in 15:49:45 #15 1.767 _CACHE[Path(sys.executable)] = PythonInfo() 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/discovery/py_info.py", line 100, in init 15:49:45 #15 1.767 self.distutils_install = self._distutils_install().copy() 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/discovery/py_info.py", line 191, in _distutils_install 15:49:45 #15 1.767 i.finalize_options() 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/command/install.py", line 57, in finalize_options 15:49:45 #15 1.767 super().finalize_options() 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/install.py", line 407, in finalize_options 15:49:45 #15 1.767 'dist_fullname': self.distribution.get_fullname(), 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 266, in get_fullname 15:49:45 #15 1.767 return _distribution_fullname(self.get_name(), self.get_version()) 15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname 15:49:45 #15 1.767 canonicalize_version(version, strip_trailing_zero=False), 15:49:45 #15 1.767 TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero' 15:49:45 #15 1.767 15:49:45 #15 1.769 Failed to create virtual environment. 15:49:45 #15 ERROR: process "/bin/sh -c pipenv install --python 3.9" did not complete successfully: exit code: 1 15:49:45 ------ 15:49:45 > [11/12] RUN pipenv install --python 3.9: 15:49:45 1.767 super().finalize_options() 15:49:45 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/install.py", line 407, in finalize_options 15:49:45 1.767 'dist_fullname': self.distribution.get_fullname(), 15:49:45 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 266, in get_fullname 15:49:45 1.767 return _distribution_fullname(self.get_name(), self.get_version()) 15:49:45 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname 15:49:45 1.767 canonicalize_version(version, strip_trailing_zero=False), 15:49:45 1.767 TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero' 15:49:45 1.767 15:49:45 1.769 Failed to create virtual environment.

Steps to replicate

Run the following docker script on a t2.micro AWS EC2 instance built using the following AMI:

amzn2-ami-hvm-2.0.20230926.0-x86_64-gp2 ami-0db2efede9f8bb581 Amazon Linux 2 AMI 2.0.20230926.0 x86_64 HVM gp2

###############DOCKER SCRIPT START########################### ############################################################ FROM ubuntu:focal

ENV TZ=Europe/London \ DEBIAN_FRONTEND=noninteractive \ IS_DOCKER=true \ PYTHONPATH=/code

RUN apt-get -y update \ && apt-get -y install \ awscli \ ca-certificates \ gnupg \ gnupg1 \ gnupg2 \ libcairo2 \ libcairo2-dev \ libgeos-dev \ libpq-dev \ libgirepository1.0-dev \ libspatialindex-dev \ osm2pgsql \ python3.9-dev \ && echo -n 'deb [ arch=amd64 ] https://downloads.skewed.de/apt focal main' >> /etc/apt/sources.list \ && apt-key adv --keyserver keys.openpgp.org --recv-key 612DEFB798507F25 \ && apt-get -y update \ && apt-get install python3-graph-tool=2.45 -y

RUN apt-get install -y curl RUN curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python3.9

RUN pipenv install --python 3.9

###############DOCKER SCRIPT END############################# ############################################################

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

Tried running this but it didn't work:

18:07:29 #15 [11/12] RUN pipenv --support | xclip 18:07:29 #15 0.561 /bin/sh: 1: xclip: not found 18:07:31 #15 ERROR: process "/bin/sh -c pipenv --support | xclip" did not complete successfully: exit code: 127

[Whylickspittle]

TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero' I meet the same issue

[evallesp]

Here's more info: https://github.com/pypa/setuptools/issues/4483

[matteius]

Here's more info: https://github.com/pypa/setuptools/issues/4483

I suggest following the guidance in the setuptools issue as my current understanding is your issue is actually occurring in setuptools/packaging and should not be considered a pipenv bug.

[viktorvorobev]

Hello! I've faced the same issue and to me, it helped to add the following, before actually creating pipenv environment:

RUN python3 -m pip install --upgrade --no-cache-dir packaging==22.0
RUN python3 -m pipenv sync --dev

Apparently, packaging is the dependency of virtualenv, while virtualenv is the dependency of pipenv.

I'm using Python 3.8 (because of reasons, don't blame me). With this version and pipenv==2024.0.1:

• if using packaging==21.3, pipenv fails with TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
• if using packaging==22.0, everything seems to be working

Your results may vary depending on used versions of python and all other stuff.

Hopefully that helps!

[matteius]

Should pipenv add packaging>=22.0 to our pypoject.toml dependencies? Would that solve the issue @viktorvorobev ?

[viktorvorobev]

@matteius I believe it could, yes, but it is still a mystery to me, why on Earth, the optional dependency of the virtualenv can break pipenv this way. Guess, you should know this better than me :)

Anyway, I'm happy with my workaround for now, so you do you. My Dockerfile is a mess of its own, so there is no harm in a new line, and for some reason I don't face this issue with newer versions of python or without a docker.

Maybe you should just keep this issue open for the time being, and observe the impact, I don't know ¯\_(ツ)_/¯