cimg/python:3.9.15 + pipenv uses Python 3.10.6

blimmer commented 2 years ago

I use pipenv to manage dependencies for a project. With the new version of cimg/python:3.9.15 (sha256:b74e50b094007f94d474df2707b6ce60b7c438eabfce64561a4305ee4a6a72d7), pipenv uses version 3.10.6 of Python (vs. the expected 3.9.15 version).

> docker pull cimg/python:3.9.15
3.9.15: Pulling from cimg/python
Digest: sha256:b74e50b094007f94d474df2707b6ce60b7c438eabfce64561a4305ee4a6a72d7
Status: Image is up to date for cimg/python:3.9.15
docker.io/cimg/python:3.9.15
> docker run -it --rm --entrypoint bash cimg/python:3.9.15
circleci@8a86bf0a8874:~/project$ echo $PATH
/home/circleci/.pyenv/shims:/home/circleci/.pyenv/bin:/home/circleci/.poetry/bin:/home/circleci/bin:/home/circleci/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
circleci@8a86bf0a8874:~/project$ pip install pipenv
Requirement already satisfied: pipenv in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (2022.10.25)
Requirement already satisfied: virtualenv in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from pipenv) (20.16.6)
Requirement already satisfied: virtualenv-clone>=0.2.5 in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from pipenv) (0.5.7)
Requirement already satisfied: certifi in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from pipenv) (2022.9.24)
Requirement already satisfied: setuptools>=36.2.1 in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from pipenv) (58.1.0)
Requirement already satisfied: platformdirs<3,>=2.4 in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from virtualenv->pipenv) (2.5.2)
Requirement already satisfied: filelock<4,>=3.4.1 in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from virtualenv->pipenv) (3.8.0)
Requirement already satisfied: distlib<1,>=0.3.6 in /home/circleci/.pyenv/versions/3.9.15/lib/python3.9/site-packages (from virtualenv->pipenv) (0.3.6)
circleci@8a86bf0a8874:~/project$ pipenv --version
pipenv, version 2022.10.25
circleci@8a86bf0a8874:~/project$ pipenv install
Creating a virtualenv for this project...
Pipfile: /home/circleci/project/Pipfile
Using /usr/bin/python3 (3.10.6) to create virtualenv...
⠦ Creating virtual environment...created virtual environment CPython3.10.6.final.0-64 in 2302ms
  creator Venv(dest=/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Posix)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q
Creating a Pipfile for this project...
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Updated Pipfile.lock (a36a5392bb1e8bbc06bfaa0761e52593cf2d83b486696bf54667ba8da616c839)!
Installing dependencies from Pipfile.lock (16c839)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
circleci@8a86bf0a8874:~/project$ which python
/home/circleci/.pyenv/shims/python
circleci@8a86bf0a8874:~/project$ python --version
Python 3.9.15
circleci@8a86bf0a8874:~/project$ which python3
/home/circleci/.pyenv/shims/python3
circleci@8a86bf0a8874:~/project$ python3 --version
Python 3.9.15
circleci@8a86bf0a8874:~/project$ /usr/bin/python3 --version
Python 3.10.6

The key from these logs is:

Using /usr/bin/python3 (3.10.6) to create virtualenv...

Which is unexpected.

This did not occur with the cimg/python:3.9.14 image:

> docker pull cimg/python:3.9.14
3.9.14: Pulling from cimg/python
Digest: sha256:980d486c5c91d243893b59a9471ad78c2f720cbadf8e87310f9dc8049cb71b98
Status: Image is up to date for cimg/python:3.9.14
docker.io/cimg/python:3.9.14
> docker run -it --rm --entrypoint bash cimg/python:3.9.14
circleci@8014e15a91ae:~/project$ echo $PATH
/home/circleci/.pyenv/shims:/home/circleci/.pyenv/bin:/home/circleci/.poetry/bin:/home/circleci/bin:/home/circleci/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
circleci@8014e15a91ae:~/project$ pip install pipenv
Requirement already satisfied: pipenv in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (2022.9.24)
Requirement already satisfied: setuptools>=36.2.1 in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from pipenv) (58.1.0)
Requirement already satisfied: certifi in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from pipenv) (2022.9.24)
Requirement already satisfied: virtualenv in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from pipenv) (20.16.5)
Requirement already satisfied: virtualenv-clone>=0.2.5 in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from pipenv) (0.5.7)
Requirement already satisfied: platformdirs<3,>=2.4 in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from virtualenv->pipenv) (2.5.2)
Requirement already satisfied: filelock<4,>=3.4.1 in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from virtualenv->pipenv) (3.8.0)
Requirement already satisfied: distlib<1,>=0.3.5 in /home/circleci/.pyenv/versions/3.9.14/lib/python3.9/site-packages (from virtualenv->pipenv) (0.3.6)

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: pip install --upgrade pip
circleci@8014e15a91ae:~/project$ pipenv --version
pipenv, version 2022.9.24
circleci@8014e15a91ae:~/project$ pipenv install
Creating a virtualenv for this project...
Pipfile: /home/circleci/project/Pipfile
Using /home/circleci/.pyenv/versions/3.9.14/bin/python3.9 (3.9.14) to create virtualenv...
⠏ Creating virtual environment...created virtual environment CPython3.9.14.final.0-64 in 1703ms
  creator CPython3Posix(dest=/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.3.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q
Creating a Pipfile for this project...
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Updated Pipfile.lock (16c839)!
Installing dependencies from Pipfile.lock (16c839)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
circleci@8014e15a91ae:~/project$ which python
/home/circleci/.pyenv/shims/python
circleci@8014e15a91ae:~/project$ python --version
Python 3.9.14
circleci@8014e15a91ae:~/project$ which python3
/home/circleci/.pyenv/shims/python3
circleci@8014e15a91ae:~/project$ python3 --version
Python 3.9.14
circleci@8014e15a91ae:~/project$ /usr/bin/python3 --version
Python 3.8.10

It used the correct version for the pipenv:

Using /home/circleci/.pyenv/versions/3.9.14/bin/python3.9 (3.9.14) to create virtualenv...

But, /usr/bin/python3 is oddly 3.8.10 on this image, as well...

circleci@8014e15a91ae:~/project$ /usr/bin/python3 --version
Python 3.8.10

In my opinion, every possible incantation of running python in the cimg/python images should always use the version specified in the tag.

JalexChen commented 2 years ago

Hi @blimmer - thanks for bringing this up.

seems the issue is a combination of the pyenv shim being passed as a value to pipenv, how pipenv looks for the python version, and virtualenv cache

pipenv --rm then pipenv install resets the cache and builds with the correct version the way pipenv looks and uses python is to use the highest version here

best practice would be to specify which version of python to use e.g pipenv install --python=3.9.15, but i agree it should match the tag. if anyone wants to change the version, they can always specify

blimmer commented 2 years ago

@JalexChen - thank you! It looks like this change wasn't published to DockerHub. What're your thoughts about releasing this?

JalexChen commented 2 years ago

Hey @blimmer - general policy is to only respin if there are security issues. Since there is a wrokaround, these changes will be reflected for the next release.

CircleCI-Public / cimg-python

cimg/python:3.9.15 + pipenv uses Python 3.10.6 #167