Staging fails with vendored dependencies in > 1.8.9

[mhaas]

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{"name":"","build":"","support":"","version":0,"description":"SAP BTP Cloud Foundry environment","authorization_endpoint":"https://login.cf.sap.hana.ondemand.com","token_endpoint":"https://uaa.cf.sap.hana.ondemand.com","min_cli_version":null,"min_recommended_cli_version":null,"app_ssh_endpoint":"ssh.cf.sap.hana.ondemand.com:2222","app_ssh_host_key_fingerprint":"SYWfpJUXGL2T+b8bkK15zppx/fZ08Wc9/iV8bxMLqWI","app_ssh_oauth_client":"ssh-proxy","doppler_logging_endpoint":"wss://doppler.cf.sap.hana.ondemand.com:443","api_version":"2.207.0","osbapi_version":"2.15","user":"9faa224f-3d69-4064-9c16-69e758b19d6a"}

➜ cf version 
cf version 8.6.1+b5a352a.2023-02-27

What version of the buildpack you are using?

Working: 1.8.9 and older Broken: 1.8.10 and newer

If you were attempting to accomplish a task, what was it you were attempting to do?

I am pushing an app with vendored sdist dependencies. Pip attempts to build a wheel of these dependencies during installation which fails because the wheel package is not installed. This probably happens due to use of --build-isolation, which breaks PEP 517 packages (see #171).

What did you expect to happen?

Staging should work without an error, like it does for non-vendored apps with the same set of dependencies.

I am providing steps to reproduce here: https://github.com/mhaas/cf-python-buildpack-vendoring-issue

What was the actual behavior?

Staging fails with this error:

-----> Running Pip Install (Vendored)
   Using the pip --no-build-isolation flag since it is available
   python -m pip install -r /tmp/app/requirements.txt --ignore-installed --exists-action=w --src=/tmp/contents3563999740/deps/0/src --no-index --find-links=file:///tmp/app/vendor --disable-pip-version-check --no-warn-script-location --no-build-isolation
   Looking in links: file:///tmp/app/vendor
   Processing ./vendor/oss2-2.18.1.tar.gz (from -r /tmp/app/requirements.txt (line 1))
   Preparing metadata (pyproject.toml): started
   Preparing metadata (pyproject.toml): finished with status 'error'
   error: subprocess-exited-with-error

   × Preparing metadata (pyproject.toml) did not run successfully.
   │ exit code: 1
   ╰─> [13 lines of output]
   running dist_info
   creating /tmp/pip-modern-metadata-r605vwxw/oss2.egg-info
   writing /tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/PKG-INFO
   writing dependency_links to /tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/dependency_links.txt
   writing requirements to /tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/requires.txt
   writing top-level names to /tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/top_level.txt
   writing manifest file '/tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/SOURCES.txt'
   reading manifest file '/tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/SOURCES.txt'
   reading manifest template 'MANIFEST.in'
   adding license file 'LICENSE'
   writing manifest file '/tmp/pip-modern-metadata-r605vwxw/oss2.egg-info/SOURCES.txt'
   creating '/tmp/pip-modern-metadata-r605vwxw/oss2-2.18.1.dist-info'
   error: invalid command 'bdist_wheel'
   [end of output]

Please confirm where necessary:

• [x] I have included a log output
• [x] My log includes an error message
• [x] I have included steps for reproduction - see this repo: https://github.com/mhaas/cf-python-buildpack-vendoring-issue

[bondurantjordan]

I'm having the same issue; all vendored tarballs fail with error "invalid command bdist_wheel". Attempted workarounds including:

1. adding wheel to requirements.txt and pip installing all tar.gz files in start command. Dependencies cause issues and the start command gets really long
2. Creating wheel files from tar.gz, but it is hard to create a tarball for cflinuxfs3 on a windows machine. Could try creating each whl file with a Docker image?

[mhaas]

I have updated my description above to reference 1.8.9 instead of 1.8.10. I am not sure why 1.8.10 seemed to work for me; in my testing with the terraform CF provider, I observed that the buildpack version was not applied correctly.

I found a hint in my logs when running 1.8.9:

   2023-08-25T12:16:31.39+0200 [STG/0] ERR DEPRECATION: oss2 is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

This message explains why 1.8.9 works and later versions break: 1.8.9 is the last version that comes with pip 23.0 according to the changelog

[robdimsdale]

Thanks for reporting this and sorry this is happening.

I always get a bit confused with python build tools packages, like wheel and setuptools, and how they interact with the various PEPs and setup.py, etc.

I think the desired solution here is that the Python Buildpack should include the wheel package and ensure it is available during application staging.

Does that sound right to you? Do you see a different path forward?

[mhaas]

Hi @robdimsdale,

yeah, I think adding wheel and setuptools would be a good fix/workaround.

I think with PEP-517, it is hard to implemented vendoring correctly. The basic problem is that the vendor/ directory would need to include the build-time dependencies specified by a PEP-517-ified sdist. I am not sure if the ecosystem (e.g. pip) currently handles this.

So, for the short term, adding wheel and setuptools is a reasonable heuristic to make most (?) packages work.

[dominikmueller]

We're having the same issue with vendored dependencies. Are there any plans to fix this in a future release or should it already be fixed?

[millsks]

Similar issues. Is there a workaround that can be used until it is fixed?

[arjun024]

https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.8.26 should fix the issue. Feel free to reopen if the issue persists