search

CircleCI-Public / cimg-python

The Next-Gen CircleCI Python Docker Convenience Image.
https://circleci.com/developer/images/image/cimg/python
MIT License
33 stars 33 forks source link

Bug Report: 'bad interpreter: No such file or directory' with new cimg/python:3.8.20 #247

Closed jdstrand closed 2 months ago

jdstrand commented 2 months ago

Describe the bug My python CI jobs started failing with bad interpreter: No such file or directory when using image: cimg/python:3.8. I see from https://circleci.com/developer/images/image/cimg/python that 3.8 was recently updated to 3.8.20. Specifying image: cimg/python:3.8.19 resolves the issue.

Error output for Install python components for prod step:

Installing to '.venv'
Collecting aiohappyeyeballs==2.3.5 (from -r ./requirements.txt (line 1))
  Downloading aiohappyeyeballs-2.3.5-py3-none-any.whl.metadata (5.8 kB)
Collecting aiohttp==3.10.3 (from -r ./requirements.txt (line 2))
  Downloading aiohttp-3.10.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)
Collecting aiosignal==1.3.1 (from -r ./requirements.txt (line 3))
  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
Collecting async-timeout==4.0.3 (from -r ./requirements.txt (line 4))
  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
Collecting attrs==23.2.0 (from -r ./requirements.txt (line 5))
  Downloading attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting blinker==1.8.2 (from -r ./requirements.txt (line 6))
  Downloading blinker-1.8.2-py3-none-any.whl.metadata (1.6 kB)
Collecting cachetools==5.5.0 (from -r ./requirements.txt (line 7))
  Downloading cachetools-5.5.0-py3-none-any.whl.metadata (5.3 kB)
Collecting certifi==2024.7.4 (from -r ./requirements.txt (line 8))
  Downloading certifi-2024.7.4-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.3.2 (from -r ./requirements.txt (line 9))
  Using cached charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Requirement already satisfied: click==8.1.7 in /home/circleci/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 10)) (8.1.7)
Collecting flask==3.0.3 (from -r ./requirements.txt (line 11))
  Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)
Collecting frozenlist==1.4.1 (from -r ./requirements.txt (line 12))
  Downloading frozenlist-1.4.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting geoip2==4.8.0 (from -r ./requirements.txt (line 13))
  Downloading geoip2-4.8.0-py2.py3-none-any.whl.metadata (18 kB)
Collecting google-api-core==2.19.2 (from -r ./requirements.txt (line 14))
  Downloading google_api_core-2.19.2-py3-none-any.whl.metadata (2.7 kB)
Collecting google-api-python-client==2.141.0 (from -r ./requirements.txt (line 15))
  Downloading google_api_python_client-2.141.0-py2.py3-none-any.whl.metadata (6.7 kB)
Collecting google-auth==2.33.0 (from -r ./requirements.txt (line 16))
  Downloading google_auth-2.33.0-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting google-auth-httplib2==0.2.0 (from -r ./requirements.txt (line 17))
  Downloading google_auth_httplib2-0.2.0-py2.py3-none-any.whl.metadata (2.2 kB)
Collecting googleapis-common-protos==1.65.0 (from -r ./requirements.txt (line 18))
  Downloading googleapis_common_protos-1.65.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting gunicorn==22.0.0 (from -r ./requirements.txt (line 19))
  Downloading gunicorn-22.0.0-py3-none-any.whl.metadata (4.4 kB)
Collecting httplib2==0.22.0 (from -r ./requirements.txt (line 20))
  Downloading httplib2-0.22.0-py3-none-any.whl.metadata (2.6 kB)
Collecting idna==3.7 (from -r ./requirements.txt (line 21))
  Downloading idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting importlib-metadata==8.0.0 (from -r ./requirements.txt (line 22))
  Downloading importlib_metadata-8.0.0-py3-none-any.whl.metadata (4.6 kB)
Collecting importlib-resources==6.4.0 (from -r ./requirements.txt (line 23))
  Downloading importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB)
Collecting influxdb3-python==0.6.1 (from -r ./requirements.txt (line 24))
  Downloading influxdb3_python-0.6.1-py3-none-any.whl.metadata (8.4 kB)
Collecting itsdangerous==2.2.0 (from -r ./requirements.txt (line 25))
  Downloading itsdangerous-2.2.0-py3-none-any.whl.metadata (1.9 kB)
Collecting jinja2==3.1.4 (from -r ./requirements.txt (line 26))
  Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting jsonschema==4.22.0 (from -r ./requirements.txt (line 27))
  Downloading jsonschema-4.22.0-py3-none-any.whl.metadata (8.2 kB)
Collecting jsonschema-specifications==2023.12.1 (from -r ./requirements.txt (line 28))
  Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting MarkupSafe==2.1.5 (from -r ./requirements.txt (line 29))
  Downloading MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting maxminddb==2.6.2 (from -r ./requirements.txt (line 30))
  Downloading maxminddb-2.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.0 kB)
Collecting multidict==6.0.5 (from -r ./requirements.txt (line 31))
  Downloading multidict-6.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting numpy==1.24.4 (from -r ./requirements.txt (line 32))
  Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Requirement already satisfied: packaging==24.1 in /home/circleci/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 33)) (24.1)
Collecting pkgutil-resolve-name==1.3.10 (from -r ./requirements.txt (line 34))
  Downloading pkgutil_resolve_name-1.3.10-py3-none-any.whl.metadata (624 bytes)
Collecting proto-plus==1.24.0 (from -r ./requirements.txt (line 35))
  Downloading proto_plus-1.24.0-py3-none-any.whl.metadata (2.2 kB)
Collecting protobuf==5.28.0 (from -r ./requirements.txt (line 36))
  Downloading protobuf-5.28.0-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Collecting pyarrow==16.1.0 (from -r ./requirements.txt (line 37))
  Downloading pyarrow-16.1.0-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Collecting pyasn1==0.6.0 (from -r ./requirements.txt (line 38))
  Downloading pyasn1-0.6.0-py2.py3-none-any.whl.metadata (8.3 kB)
Collecting pyasn1-modules==0.4.0 (from -r ./requirements.txt (line 39))
  Downloading pyasn1_modules-0.4.0-py3-none-any.whl.metadata (3.4 kB)
Collecting pyparsing==3.1.4 (from -r ./requirements.txt (line 40))
  Downloading pyparsing-3.1.4-py3-none-any.whl.metadata (5.1 kB)
Collecting python-dateutil==2.9.0.post0 (from -r ./requirements.txt (line 41))
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting PyYAML==6.0.1 (from -r ./requirements.txt (line 42))
  Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting reactivex==4.0.4 (from -r ./requirements.txt (line 43))
  Downloading reactivex-4.0.4-py3-none-any.whl.metadata (5.5 kB)
Collecting referencing==0.35.1 (from -r ./requirements.txt (line 44))
  Downloading referencing-0.35.1-py3-none-any.whl.metadata (2.8 kB)
Collecting requests==2.32.3 (from -r ./requirements.txt (line 45))
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting rpds-py==0.18.1 (from -r ./requirements.txt (line 46))
  Downloading rpds_py-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting rsa==4.9 (from -r ./requirements.txt (line 47))
  Downloading rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Collecting six==1.16.0 (from -r ./requirements.txt (line 48))
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting typing-extensions==4.12.2 (from -r ./requirements.txt (line 49))
  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting uritemplate==4.1.1 (from -r ./requirements.txt (line 50))
  Downloading uritemplate-4.1.1-py2.py3-none-any.whl.metadata (2.9 kB)
Collecting urllib3==2.2.2 (from -r ./requirements.txt (line 51))
  Using cached urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB)
Collecting werkzeug==3.0.3 (from -r ./requirements.txt (line 52))
  Downloading werkzeug-3.0.3-py3-none-any.whl.metadata (3.7 kB)
Collecting yarl==1.9.4 (from -r ./requirements.txt (line 53))
  Downloading yarl-1.9.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting zipp==3.19.2 (from -r ./requirements.txt (line 54))
  Downloading zipp-3.19.2-py3-none-any.whl.metadata (3.6 kB)
Requirement already satisfied: setuptools>=60.0.0 in /home/circleci/.pyenv/versions/3.8.20/lib/python3.8/site-packages (from geoip2==4.8.0->-r ./requirements.txt (line 13)) (74.1.2)
Downloading aiohappyeyeballs-2.3.5-py3-none-any.whl (12 kB)
Downloading aiohttp-3.10.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 142.5 MB/s eta 0:00:00

Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Downloading attrs-23.2.0-py3-none-any.whl (60 kB)
Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB)
Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB)
Downloading certifi-2024.7.4-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)
Downloading flask-3.0.3-py3-none-any.whl (101 kB)
Downloading frozenlist-1.4.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (240 kB)
Downloading geoip2-4.8.0-py2.py3-none-any.whl (27 kB)
Downloading google_api_core-2.19.2-py3-none-any.whl (139 kB)
Downloading google_api_python_client-2.141.0-py2.py3-none-any.whl (12.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 211.7 MB/s eta 0:00:00

Downloading google_auth-2.33.0-py2.py3-none-any.whl (200 kB)
Downloading google_auth_httplib2-0.2.0-py2.py3-none-any.whl (9.3 kB)
Downloading googleapis_common_protos-1.65.0-py2.py3-none-any.whl (220 kB)
Downloading gunicorn-22.0.0-py3-none-any.whl (84 kB)
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
Downloading idna-3.7-py3-none-any.whl (66 kB)
Downloading importlib_metadata-8.0.0-py3-none-any.whl (24 kB)
Downloading importlib_resources-6.4.0-py3-none-any.whl (38 kB)
Downloading influxdb3_python-0.6.1-py3-none-any.whl (73 kB)
Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
Downloading jsonschema-4.22.0-py3-none-any.whl (88 kB)
Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Downloading MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26 kB)
Downloading maxminddb-2.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (88 kB)
Downloading multidict-6.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)
Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 190.9 MB/s eta 0:00:00

Downloading pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Downloading proto_plus-1.24.0-py3-none-any.whl (50 kB)
Downloading protobuf-5.28.0-cp38-abi3-manylinux2014_x86_64.whl (316 kB)
Downloading pyarrow-16.1.0-cp38-cp38-manylinux_2_28_x86_64.whl (40.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 MB 97.0 MB/s eta 0:00:00

Downloading pyasn1-0.6.0-py2.py3-none-any.whl (85 kB)
Downloading pyasn1_modules-0.4.0-py3-none-any.whl (181 kB)
Downloading pyparsing-3.1.4-py3-none-any.whl (104 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (736 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 736.6/736.6 kB 38.3 MB/s eta 0:00:00

Downloading reactivex-4.0.4-py3-none-any.whl (217 kB)
Downloading referencing-0.35.1-py3-none-any.whl (26 kB)
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Downloading rpds_py-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 103.7 MB/s eta 0:00:00

Downloading rsa-4.9-py3-none-any.whl (34 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Using cached urllib3-2.2.2-py3-none-any.whl (121 kB)
Downloading werkzeug-3.0.3-py3-none-any.whl (227 kB)
Downloading yarl-1.9.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (308 kB)
Downloading zipp-3.19.2-py3-none-any.whl (9.0 kB)
Installing collected packages: zipp, urllib3, uritemplate, typing-extensions, six, rpds-py, PyYAML, pyparsing, pyasn1, protobuf, pkgutil-resolve-name, numpy, multidict, maxminddb, MarkupSafe, itsdangerous, idna, gunicorn, frozenlist, charset-normalizer, certifi, cachetools, blinker, attrs, async-timeout, aiohappyeyeballs, yarl, werkzeug, rsa, requests, referencing, reactivex, python-dateutil, pyasn1-modules, pyarrow, proto-plus, jinja2, importlib-resources, importlib-metadata, httplib2, googleapis-common-protos, aiosignal, jsonschema-specifications, influxdb3-python, google-auth, flask, aiohttp, jsonschema, google-auth-httplib2, google-api-core, geoip2, google-api-python-client
  Attempting uninstall: certifi
    Found existing installation: certifi 2024.8.30
    Uninstalling certifi-2024.8.30:
      Successfully uninstalled certifi-2024.8.30
Successfully installed MarkupSafe-2.1.5 PyYAML-6.0.1 aiohappyeyeballs-2.3.5 aiohttp-3.10.3 aiosignal-1.3.1 async-timeout-4.0.3 attrs-23.2.0 blinker-1.8.2 cachetools-5.5.0 certifi-2024.7.4 charset-normalizer-3.3.2 flask-3.0.3 frozenlist-1.4.1 geoip2-4.8.0 google-api-core-2.19.2 google-api-python-client-2.141.0 google-auth-2.33.0 google-auth-httplib2-0.2.0 googleapis-common-protos-1.65.0 gunicorn-22.0.0 httplib2-0.22.0 idna-3.7 importlib-metadata-8.0.0 importlib-resources-6.4.0 influxdb3-python-0.6.1 itsdangerous-2.2.0 jinja2-3.1.4 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 maxminddb-2.6.2 multidict-6.0.5 numpy-1.24.4 pkgutil-resolve-name-1.3.10 proto-plus-1.24.0 protobuf-5.28.0 pyarrow-16.1.0 pyasn1-0.6.0 pyasn1-modules-0.4.0 pyparsing-3.1.4 python-dateutil-2.9.0.post0 reactivex-4.0.4 referencing-0.35.1 requests-2.32.3 rpds-py-0.18.1 rsa-4.9 six-1.16.0 typing-extensions-4.12.2 uritemplate-4.1.1 urllib3-2.2.2 werkzeug-3.0.3 yarl-1.9.4 zipp-3.19.2

Installed to ./.venv. To use, run '. ./.venv/bin/activate'
/bin/bash: /home/circleci/project/collectors/.venv/bin/pip: /home/circleci/project/collectors/.venv/bin/python3: bad interpreter: No such file or directory

Exited with code exit status 126

To Reproduce Please provide steps to reproduce the behavior, such as a sample job or config file.

Example job file:

version: 2.1

commands:
  create_concatenated_requirements:
    description: "Concatenate all requirements*.txt files into a single file for checksumming."
    steps:
      - run:
          name: "Combine requirements*.txt files to a single file"
          command: |
            cat ./collectors/requirements*.txt > ./collectors/combined-requirements.txt

  # while 'make install-venv' only looks at requirements.txt, we want to
  # additionally install 'coverage' from ./requirements_dev.txt so we want to
  # use the combined-requirements.txt checksum as a key
  cache_restore:
    description: Restore python cache for prod
    steps:
      - create_concatenated_requirements
      - restore_cache:
          name: Restore python cache
          key: deps1.0-{{ .Branch }}-{{ checksum "collectors/combined-requirements.txt" }}

  cache_save:
    description: Save python cache for prod
    steps:
      - save_cache:
          name: Save python cache
          key: deps1.0-{{ .Branch }}-{{ checksum "collectors/combined-requirements.txt" }}
          paths:
            - "collectors/.venv"

  python_components:
    description: Install python components for prod
    steps:
      - run:
          name: Install python components
          command: |
            cd ./collectors
            make install-venv
            # install coverage separately since we don't want it in prod
            . .venv/bin/activate
            pip install $(grep '^coverage=' ./requirements_dev.txt)

jobs:
  tests:
    docker:
      - image: cimg/python:3.8
    steps:
      - checkout
      - cache_restore
      - python_components
      - run:
          name: Running coverage
          command: |
            cd ./collectors
            . .venv/bin/activate
            make coverage
            make coverage-report
      - cache_save

# workflows can run in parallel
workflows:
  version: 2
  ci:
    jobs:
      - tests

make install-venv does:

install-venv:
    @if ! test -d "./$(VENV)" ; then \
        echo "Running 'python3 -m venv $(VENV)'" ; \
        python3 -m venv $(VENV) ; \
    fi
    @if test -z "$(VIRTUAL_ENV)" ; then \
        echo "Installing to '$(VENV)'" ; \
        . ./$(VENV)/bin/activate ; \
        pip install -r ./requirements.txt ; \
        echo "\nInstalled to ./$(VENV). To use, run '. ./$(VENV)/bin/activate'" ; \
    else \
        echo "Updating '$(VENV)'" ; \
        pip install -r ./requirements.txt ; \
        echo "\nUpdated ./$(VENV)." ; \
    fi

Expected behavior Normally, this all works fine and tests are run.

Workarounds Are there any current workarounds for this bug that can be used currently?

Use:

  tests:
    docker:
      - image: cimg/python:3.8.19

Screenshots and Build Links If possible, add screenshots and links to jobs to help explain your problem.

Additional context Add any other context about the problem here.

merwan commented 2 months ago

I had a similar issue with the Python 3.10 base image. I invalidated the CircleCI cache because it referenced an non-existent version of Python. To make it easier to invalidate the cache, I configure the cache key with an environment variable so that I do not have to commit a change to the config.yml file. For example:

key: deps-{{ .Environment.CACHE_VERSION }}-{{ .Branch }}-{{ checksum "collectors/combined-requirements.txt" }}
jdstrand commented 2 months ago

Ah, yes, I was wondering if it had something to do with the cache. Thanks for the tip! I'm going to close this as it has to do with my setup of the cache.

jdstrand commented 2 months ago

In case it is useful to others, I decided to fix this for my use case in the following way:

commands:
  create_concatenated_requirements:
    description: "Concatenate all requirements*.txt files into a single file for checksumming."
    steps:
      - run:
          name: "Combine requirements*.txt files to a single file"
          command: |
            cat ./collectors/requirements*.txt > ./collectors/combined-requirements.txt
            # detect if the python (micro) version changes
            echo "# python3 --version: $(python3 --version)" >> ./collectors/combined-requirements.txt
            # for debugging
            #echo "$ cat ./collectors/combined-requirements.txt"
            #cat ./collectors/combined-requirements.txt

This works because in my case I am using the checksum of ./collectors/combined-requirements.txt as part of my key (ie, key: deps1.0-{{ .Branch }}-{{ checksum "collectors/combined-requirements.txt" }}) and while I am not using ./collectors/combined-requirements.txt with pip, requirements files support comments so the file remains compatible with it.