Open jellehelsen opened 1 year ago
I think there's a few things going on here.
Firstly, we don't ship the python distribution that is provided by canonical - we compile from source (see here). So you'd actually need to look at the upstream source.
That being said, I'm assuming that the upstream cpython source has the fix which means we would expect to see that fix in the buildpack. The Python buildpacks provide the latest two patch releases of all supported minors.
Which then leads me to ask what version of the python buildpack are you actually using? You can see this information in the output of the build (e.g. pack build
).
Once we have the version of the buildpack we can talk about specific versions of python that are present.
Relevant part of the build log:
[builder] Paketo Buildpack for CPython 1.9.0
[builder] Resolving CPython version
[builder] Candidate version sources (in priority order):
[builder] pyproject.toml -> "^3.11"
[builder] -> ""
[builder] <unknown> -> ""
[builder]
[builder] Selected CPython version (using pyproject.toml): 3.11.5
[builder]
[builder] Executing build process
[builder] Installing CPython 3.11.5
[builder] Completed in 3.331s
Is this the info you need?
Yup, that's helpful, thanks. You also answered my next question I was going to ask - the version of python available.
So you can see that you're using Python 3.11.5 - the latest available.
I think in order to track down this CVE report you'll need to look at your scanner to determine which file on the filesystem is actually contributing to that scan.
We see a lot of false positives with scanners. I'm not necessarily saying (yet) that this is one of them, but I don't think the issue is an out of date version of Python.
From inside the built container, showing the bad version.
cnb@2a0a352d462c:/$ ls -l /layers/paketo-buildpacks_cpython/cpython/lib/python3.11/site-packages/
total 32
-rw-r--r-- 1 1001 cnb 119 Jan 1 1980 README.txt
drwxr-xr-x 3 1001 cnb 4096 Jan 1 1980 _distutils_hack
-rw-r--r-- 1 1001 cnb 151 Jan 1 1980 distutils-precedence.pth
drwxr-xr-x 5 1001 cnb 4096 Jan 1 1980 pip
drwxr-xr-x 2 1001 cnb 4096 Jan 1 1980 pip-23.1.2.dist-info
drwxr-xr-x 5 1001 cnb 4096 Jan 1 1980 pkg_resources
drwxr-xr-x 8 1001 cnb 4096 Jan 1 1980 setuptools
drwxr-xr-x 2 1001 cnb 4096 Jan 1 1980 setuptools-65.5.0.dist-info
As it happens, I also use the poetry buildpack, and in the virtual enviroment that poetry creates the version of setuptools is way higher:
cnb@2a0a352d462c:/$ ls -l /layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/lib/python3.11/site-packages/ | grep setuptools
drwxr-xr-x 7 1001 cnb 4096 Jan 1 1980 setuptools
drwxr-xr-x 2 1001 cnb 4096 Jan 1 1980 setuptools-68.2.2.dist-info
So the vulnerable version is still installed, albeit not used.
Hope this helps.
Ah, setuptools
.
The only place in the buildpack toolchain that we explicitly interact with setup tools is in the pip
buildpack, where we download it as part of the pre-compiled pip
dependency.
When I look at the contents of the most recent pip
dependency (https://artifacts.paketo.io/pip/pip_23.2.0_noarch_ba051f2f.tgz) I see the following:
❯ ls -alh | grep setuptools
-rw-r--r-- 1 ubuntu ubuntu 2.1M Aug 2 14:11 setuptools-68.0.0.tar.gz
So if you're getting 65.5.0
then I'm guess either you're using an old version of the pip
buildpack (which is a dependency of poetry
) or it's a transitive dependency of your application
I don't agree. I'm using the latest version of the pip
buildpack (0.18.0). It being a dependency of my application does not really makes sense to me. All dependencies of my application are installed with poetry and are installed inside the poetry virtualenv. As I've shown in my previous message, the version of setuptools in the virtualenv is 68.2.2
. Only the version in the system python is wrong.
Complete builder log:
[builder] Timer: Builder started at 2023-09-21T09:29:20Z
[builder]
[builder] Paketo Buildpack for CA Certificates 3.6.3
[builder] https://github.com/paketo-buildpacks/ca-certificates
[builder] Launch Helper: Contributing to layer
[builder] Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder] Paketo Buildpack for CPython 1.9.0
[builder] Resolving CPython version
[builder] Candidate version sources (in priority order):
[builder] pyproject.toml -> "^3.11"
[builder] -> ""
[builder] <unknown> -> ""
[builder]
[builder] Selected CPython version (using pyproject.toml): 3.11.5
[builder]
[builder] Executing build process
[builder] Installing CPython 3.11.5
[builder] Completed in 3.331s
[builder]
[builder] Generating SBOM for /layers/paketo-buildpacks_cpython/cpython
[builder] Completed in 0s
[builder]
[builder]
[builder] Configuring build environment
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_cpython/cpython"
[builder] PYTHONPYCACHEPREFIX -> "/tmp"
[builder]
[builder] Configuring launch environment
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_cpython/cpython"
[builder]
[builder] Paketo Buildpack for Pip 0.18.0
[builder] Resolving Pip version
[builder] Candidate version sources (in priority order):
[builder] -> ""
[builder]
[builder] Selected Pip version (using ): 23.2.1
[builder]
[builder] Executing build process
[builder] Installing Pip 23.2.1
[builder] Completed in 10.589s
[builder]
[builder] Generating SBOM for /layers/paketo-buildpacks_pip/pip
[builder] Completed in 0s
[builder]
[builder] Configuring build environment
[builder] PIP_FIND_LINKS -> "$PIP_FIND_LINKS /layers/paketo-buildpacks_pip/pip-source"
[builder]
[builder] Configuring build environment
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_pip/pip/lib/python3.11/site-packages:$PYTHONPATH"
[builder]
[builder] Configuring launch environment
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_pip/pip/lib/python3.11/site-packages:$PYTHONPATH"
[builder]
[builder] Paketo Buildpack for Poetry 0.6.5
[builder] Resolving Poetry version
[builder] Candidate version sources (in priority order):
[builder] <unknown> -> ""
[builder]
[builder] Selected Poetry version (using <unknown>): 1.4.2
[builder]
[builder] Executing build process
[builder] Installing Poetry 1.4.2
[builder] Completed in 10.939s
[builder]
[builder] Generating SBOM for /layers/paketo-buildpacks_poetry/poetry
[builder] Completed in 0s
[builder]
[builder] Configuring build environment
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_poetry/poetry/lib/python3.11/site-packages:$PYTHONPATH"
[builder]
[builder] Configuring launch environment
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_poetry/poetry/lib/python3.11/site-packages:$PYTHONPATH"
[builder]
[builder] Paketo Buildpack for Poetry Install 0.3.17
[builder] Executing build process
[builder] Running 'POETRY_CACHE_DIR=/layers/paketo-buildpacks_poetry-install/cache POETRY_VIRTUALENVS_PATH=/layers/paketo-buildpacks_poetry-install/poetry-venv poetry install'
[builder] Creating virtualenv kubecost-function-xS3fZVNL-py3.11 in /layers/paketo-buildpacks_poetry-install/poetry-venv
[builder] Installing dependencies from lock file
[builder]
[builder] Package operations: 15 installs, 0 updates, 0 removals
[builder]
[builder] • Installing six (1.16.0)
[builder] • Installing certifi (2023.7.22)
[builder] • Installing charset-normalizer (2.1.1)
[builder] • Installing idna (3.4)
[builder] • Installing iniconfig (2.0.0)
[builder] • Installing packaging (23.1)
[builder] • Installing pluggy (1.2.0)
[builder] • Installing python-dateutil (2.8.2)
[builder] • Installing typing-extensions (4.7.1)
[builder] • Installing urllib3 (1.26.16)
[builder] • Installing datadog-api-client (2.15.0)
[builder] • Installing httpretty (1.1.4)
[builder] • Installing markdown (3.4.4)
[builder] • Installing pytest (7.4.0)
[builder] • Installing requests (2.28.1)
[builder]
[builder] Installing the current project: kubecost-function (0.1.25)
[builder] Completed in 4.786s
[builder]
[builder] Generating SBOM for /layers/paketo-buildpacks_poetry-install/poetry-venv
[builder] Completed in 17ms
[builder]
[builder] Configuring build environment
[builder] PATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/bin:$PATH"
[builder] POETRY_VIRTUALENVS_PATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv"
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/lib/python3.11/site-packages:$PYTHONPATH"
[builder]
[builder] Configuring launch environment
[builder] PATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/bin:$PATH"
[builder] POETRY_VIRTUALENVS_PATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv"
[builder] PYTHONPATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/lib/python3.11/site-packages:$PYTHONPATH"
[builder]
[builder] Paketo Buildpack for Poetry Run 0.4.21
[builder] Assigning launch processes:
[builder] web (default): poetry run kubecost
[builder]
[builder]
[builder] Paketo Buildpack for Image Labels 4.5.2
[builder] https://github.com/paketo-buildpacks/image-labels
[builder] Build Configuration:
[builder] $BP_IMAGE_LABELS arbitrary image labels
[builder] $BP_OCI_AUTHORS the org.opencontainers.image.authors image label
[builder] $BP_OCI_CREATED the org.opencontainers.image.created image label
[builder] $BP_OCI_DESCRIPTION the org.opencontainers.image.description image label
[builder] $BP_OCI_DOCUMENTATION the org.opencontainers.image.documentation image label
[builder] $BP_OCI_LICENSES the org.opencontainers.image.licenses image label
[builder] $BP_OCI_REF_NAME the org.opencontainers.image.ref.name image label
[builder] $BP_OCI_REVISION the org.opencontainers.image.revision image label
[builder] $BP_OCI_SOURCE https://github.com/vlaamsemilieumaatschappij/kubecost-function the org.opencontainers.image.source image label
[builder] $BP_OCI_TITLE the org.opencontainers.image.title image label
[builder] $BP_OCI_URL the org.opencontainers.image.url image label
[builder] $BP_OCI_VENDOR the org.opencontainers.image.vendor image label
[builder] $BP_OCI_VERSION the org.opencontainers.image.version image label
[builder] Image labels:
[builder] org.opencontainers.image.source
[builder] Timer: Builder ran for 29.790927[588](https://github.com/vlaamsemilieumaatschappij/kubecost-function/actions/runs/6259807481/job/16996552356#step:6:589)s and ended at 2023-09-21T09:29:49Z
To make things more clear I've created a test repo @ https://github.com/jellehelsen/cpython-buildpack-test . No dependencies, just cpython-3.11. The only code is print("hello world")
. Still the vulnerable setuptools version is used.
You can check the build log showing to error here.
Hi @jellehelsen, I think I see what the issue is here. As you've shown here (& verified by downloading the source distribution for python
3.11.5 from here) the offending version of setuptools
is shipped with v3.11.5 by default. There are ways to force the removal of that setuptools
version, such as invoking pip install --force-reinstall
, but as you are using poetry
(and because there isn't currently any such cleanup logic in the buildpack), v65.5.0 sticks around in the cpython
layer.
I downloaded python
v3.12 and did not find a setuptools version in the same place as in v3.11.5, so I wonder whether simply upgrading to v3.12 would solve the scanning problem.
We've identified that the source distribution of python 3.1 (1and earlier) contains an old version of setuptools with known CVEs. We build from source, so we provide that version of setuptools. We've identified that python 3.12 contains a newer version of setuptools which doesn't have that CVE.
If you zoom out a level, what we're saying is:
When I look at it that way, I don't think there's any action we should take. In general the buildpacks do not attempt to fix CVEs in upstream distributions of the dependencies. The buildpacks provide a lot of value to application developers and platform operators, but patching the upstream dependencies is outside of the scope of what we provide.
The fact that Canonical (and other re-distributors like Red Hat) have fixed this CVE in their distributions of python doesn't mean that we can take on a similar burden in the buildpacks. These organizations have far more resources available to them to test the patches they make to software they redistribute. We don't have the bandwidth to make a similar committment.
Setuptools was removed from Python 3.12 (source):
gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.
Building with Python 3.12 results in the following:
❯ docker run --rm -it --entrypoint=launcher <resultant app image SHA> ls -alh /layers/paketo-buildpacks_cpython/cpython/lib/python3.12/site-packages/
total 20K
drwxr-xr-x 4 1001 cnb 4.0K Jan 1 1980 .
drwxr-xr-x 39 1001 cnb 4.0K Jan 1 1980 ..
-rw-r--r-- 1 1001 cnb 119 Jan 1 1980 README.txt
drwxr-xr-x 5 1001 cnb 4.0K Jan 1 1980 pip
drwxr-xr-x 2 1001 cnb 4.0K Jan 1 1980 pip-23.2.1.dist-info
After quite a bit of hacking around, I was able to run pip3 install --force-reinstall setuptools
as part of the cpython build process, and it successfully upgraded the version of setuptools:
❯ docker run --rm -it --entrypoint=launcher <resultant image SHA> ls -alh /layers/paketo-buildpacks_cpython/cpython/lib/python3.10/site-packages/
total 40K
drwxr-xr-x 8 1001 cnb 4.0K Jan 1 1980 .
drwxr-xr-x 36 1001 cnb 4.0K Jan 1 1980 ..
-rw-r--r-- 1 1001 cnb 119 Jan 1 1980 README.txt
drwxr-xr-x 2 1001 cnb 4.0K Jan 1 1980 _distutils_hack
-rw-r--r-- 1 1001 cnb 151 Jan 1 1980 distutils-precedence.pth
drwxr-xr-x 5 1001 cnb 4.0K Jan 1 1980 pip
drwxr-xr-x 2 1001 cnb 4.0K Jan 1 1980 pip-23.0.1.dist-info
drwxr-xr-x 4 1001 cnb 4.0K Jan 1 1980 pkg_resources
drwxr-xr-x 7 1001 cnb 4.0K Jan 1 1980 setuptools
drwxr-xr-x 2 1001 cnb 4.0K Jan 1 1980 setuptools-68.2.2.dist-info
Unfortunately, this has two problems:
We could potentially work around both of these issues by performing the pip install --force-reinstall
logic during the compilation process of the cpython distribution, rather than during the build phase.
However, this gets into the question of what should the supported surface area of the buildpacks actually be, and I don't think we can commit to modifying the upstream source distribution.
Searching through the source of Python 3.11.5 for the string 65.5
shows us that Python 3.11.5 only contains setuptools v65.5.0
. There is no later version provided.
❯ rg -i -. '65\.5'
Lib/ensurepip/__init__.py
13:_SETUPTOOLS_VERSION = "65.5.0"
<...>
Misc/NEWS
1736: versions 22.3 and 65.5.0 respectively.
In order to get the pip3 install --force-reinstall setuptools
command working I had to rewrite the first line of the pip3
command to point to the correct location of the python3 interpreter. Otherwise you encounter the following error:
/layers/paketo-buildpacks_cpython/cpython/bin/pip3: /tmp/tmp.7dyEKmx8Dm/bin/python3.10: bad interpreter: No such file or directory
If you open pip3
(it's just a text file) you see:
#!/tmp/tmp.7dyEKmx8Dm/bin/python3.10
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
This /tmp
path is the path that the cpython distribution was compiled under as part of the compilation process of the cpython distribution itself.
In order to invoke pip3
as part of the cpython build phase, we'll have to edit this file to point to the resultant location of the python distribution on the filesystem (i.e. /layers/paketo-buildpacks_cpython/cpython/bin/python3
.
For the purposes of this investigation, I edited it during the build phase. We could/should make this change during the actual compilation if we wanted to invoke pip during the cpython build phase.
I have created a pull request to address this issue because there is no other clean way to address this CVE.
The main driver for this is to address a CVE that cannot be fixed by changing dependencies in requirement.txt files. The only way to get around this is issue is to modify the cpython layer with an inline buildpack (at build time) or to modify the container after build. Both are not good solutions.
I agree that generally buildpacks shouldn't address CVEs, but this is a case where the CVE can't be addressed without modifying the buildpack or by modifying the image created with the buildpack. No changes to requirements.txt will prevent the CVE if you are using older versions of python.
To further demonstrate that this issue can only be addressed in the cpython bulidpack, I have built the following sample app and run the Trivy scan to show the CVE. Note that the sample app I'm building has no external (pip) dependencies and it installs the default version of cpython (3.10.x), which is impacted by the CVE.
git clone https://github.com/paketo-buildpacks/samples.git
pack build no-package-manager-sample --path samples/python/no_package_manager --buildpack paketo-buildpacks/python --builder paketobuildpacks/builder-jammy-base
trivy image no-package-manager-sample --severity HIGH,CRITICAL --scanners vuln
Trivy scan output
2024-02-28T12:00:30.761-0500 INFO Vulnerability scanning is enabled
2024-02-28T12:00:30.826-0500 INFO Detected OS: ubuntu
2024-02-28T12:00:30.826-0500 INFO Detecting Ubuntu vulnerabilities...
2024-02-28T12:00:30.827-0500 INFO Number of language-specific files: 4
2024-02-28T12:00:30.827-0500 INFO Detecting gobinary vulnerabilities...
2024-02-28T12:00:30.828-0500 INFO Detecting python-pkg vulnerabilities...
no-package-manager-sample (ubuntu 22.04)
Total: 0 (HIGH: 0, CRITICAL: 0)
2024-02-28T12:00:30.829-0500 INFO Table result includes only package filenames. Use '--format json' option to get the full path to the package file.
Python (python-pkg)
Total: 1 (HIGH: 1, CRITICAL: 0)
┌───────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├───────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────┤
│ setuptools (METADATA) │ CVE-2022-40897 │ HIGH │ 65.5.0 │ 65.5.1 │ pypa-setuptools: Regular Expression Denial of Service │
│ │ │ │ │ │ (ReDoS) in package_index.py │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-40897 │
└───────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────┘
According to my trivy scan CVE-2022-40897 is still present in my newly built image, build with paketobuildpacks/builder-jammy-base. A fix was already released by Canonical a while ago (https://ubuntu.com/security/CVE-2022-40897) but the vulnerability can still be found.
According to you docs security updates are picked up from Canonical on mere hours. Why can this vulnerability can still be found?