Optionally remove setuptools using BP_CPYTHON_RM_SETUPTOOLS env var

[jericop]

Summary

This change addresses https://github.com/paketo-buildpacks/cpython/issues/579 and also cleans up broken pip executables in the cpython layer. The main issue is that the current pre-compiled installations of python include the setuptools package which has a known vulnerability. This change removes that package.

More reasons why this is helpful:

• Who is to say that future versions of setuptools in newer python releases don't suffer from the same thing. This change would just remove setuptools always to avoid the issue entirely.
• It removes unnecessary files because it is likely that end-users install their own version of setuptools for their own app dependencies. Today you end up with 2 versions of setuptools in your app image, but you really only need the one you intended to install with other dependencies.

Current workaround

When upgrading is not an option, my current workaround is to add a project.toml file with an inline build pack to remove the offending setuptools package. In the words of a fellow engineer, "YUCK." This is the last thing I want to do or encourage users to do.

Use Cases

The issue has a lot of good information in it and rightly encourages users to upgrade to a new version of python. Unfortunately this is not always an option, so this change adds a way to remove packages from the python site-packages folder in the layer as needed. It also checks for and removes broken pip executables from the installation if there are any.

The PipCleanup function will do nothing if setuptools is not installed. It will also do nothing if there are no broken pip executables.

Checklist

• [x] I have viewed, signed, and submitted the Contributor License Agreement.
• [x] I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
• [x] I have added an integration test, if necessary.
• [x] I have reviewed the styleguide for guidance on my code quality.
• [x] I'm happy with the commit history on this PR (I have rebased/squashed as needed).

[jericop]

@arjun024 Thanks for the feedback. I did read through the issue prior to creating this PR and wrote it in such a way to be in line with @robdimsdale comment.

The main driver for this is to address CVEs that cannot be fixed by changing dependencies in requirement.txt files. The only way to get around this is issue is to modify the cpython layer with an inline buildpack (at build time) or to modify the container after build. Both are not good solutions.

I will comment on his main points.

• Why not upgrade setuptools on Python 3.11.x and earlier?
  • He says we should not do this and I completely agree.
  • My solution is not a patching approach, but rather a complete fix. It avoids the need for patching by removing setup tools entirely.
  • It also prevents cache issues by implementing the fix in the cpython buildpack directly.
• Setuptools on Python 3.12
  • Python stops shipping setuptools, and this change is in line with that.
• Some additional notes
  • The pip executables do not function, so my change just removes them so there are no broken scripts/executables in the final container image.

I agree that generally buildpacks shouldn't address CVEs, but this is a case where the CVE can't be addressed without modifying the buildpack or by modifying the image created with the buildpack. No changes to requirements.txt will prevent the CVE if you are using older versions of python.

[arjun024]

• I believe the consumers of this buildpack expects CPython the way the upstream maintainers released it. Correct me if I'm wrong but with this change, an app using setuptools that currently can be built with just the cpython buildpack would be forced to add a requirements.txt, and introduce the pip-install buildpack into the build to get their app working again. If removing setuptools was the best solution for your problem, why didn't the upstream maintainers do it? I think that decision is their job and not this buildpack's. As I think about what most of our users would want, I don't think this should be the default behavior. I understand & consider your motivation to remove setuptools to be valid, and I'm ready to approve the change if it's done as an option initiated by a user-set $BP_CPYTHON_RM_SETUPTOOLS. Also, if you can convince another maintainer that removing setuptools should be the default behavior, I'll waive my objection.

• About removing the pip* files - I don't think it's worth spending time and effort during the build phase to remove them - we'd like the build to be as fast as possible and do only what's necessary. If these files are completely useless, they should be removed in the dependency compilation phase. Can you move this to a separate PR so we can have a separate discussion?

[brayanhenao]

As @arjun024 mentioned previously, I don't think the default behavior of this Buildpack is to have no setuptools. This could affect many users who use this Buildpack and force them to create a requirements.txt to add this package.

It is out of the responsibility of the Buildpack to remove this package. I think the best way to implement this feature is to use some environment variables such as BP_CPYTHON_RM_SETUPTOOLS as an alternate behavior that does not affect the default one.

The motivation for this change is clear but I feel that the expected impact outweighs the reasons to leave it as default behavior and not as an alternative.

[jericop]

Thanks for the comment @brayanhenao. In the interest of moving forward I think the environment opt-in approach is fine and I will update the code to only opt in when BP_CPYTHON_RM_SETUPTOOLS has been been set.

Here is the behavior I plan to implement:

• Only remove setuptools if BP_CPYTHON_RM_SETUPTOOLS environment variable exists.
• This would mean that the actual value of the environment variable would be ignored, meaning the following pack cli --env args would be valid to remove setuptools.
  • --env BP_CPYTHON_RM_SETUPTOOLS=
  • --env BP_CPYTHON_RM_SETUPTOOLS=true
  • --env BP_CPYTHON_RM_SETUPTOOLS=false

Any thoughts or preference on the environment variable approach?

[jericop]

I have gone ahead and implemented BP_CPYTHON_RM_SETUPTOOLS as proposed above.

[jericop]

As requested, I have removed the logic to cleanup broken pip executables and updated all unit tests to verify the changes. I have also added an integration test to verify that python works as expected when BP_CPYTHON_RM_SETUPTOOLS is set and setuptools is removed. This will confirm that the change works for all support dependencies versions and builders.

After giving it some thought and reading through some online discussions about how some packages can fall back to using setuptools when installing packages, I understand the hesitation to remove it altogether as I had originally proposed. Although I haven't encountered that myself, I can't say that I have built so many unique python containers to be sure it wouldn't be a problem. Thanks again for your feedback and patience.

[jericop]

LGTM. Thanks for the changes @jericop. I'll merge this now and make a release soon. Could you please as a follow up, document the env var & the rationale in the README and the paketo-website repo?

Apologies for forgetting to update the README. The release can wait for the readme change, which I will do later today. Thank you!