env vars not available at droplet build phase - cannot install pykcs11 as unable to set $SWIG_LIB

[lgarvey]

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

"api_version": "2.148.0", "osbapi_version": "2.15",

What version of the buildpack you are using? python-buildpack-cflinuxfs3-v1.7.13.zip

If you were attempting to accomplish a task, what was it you were attempting to do? Run a python app that has a dependency: pykcs11 which relies on SWIG

What did you expect to happen? For the app to stage and run correctly

What was the actual behavior? The build always fails with this error:

 Installing collected packages: pykcs11
            Running setup.py install for pykcs11: started
              Running setup.py install for pykcs11: finished with status 'error'
              ERROR: Command errored out with exit status 1:
               command: /tmp/contents094791880/deps/1/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y22gw937/pykcs11/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y22gw937/pykcs11/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ufpi251h/install-record.txt --single-version-externally-managed --compile
                   cwd: /tmp/pip-install-y22gw937/pykcs11/
              Complete output (13 lines):
              running install
              running build
              running build_ext
              building 'PyKCS11._LowLevel' extension
              swigging src/pykcs11.i to src/pykcs11_wrap.cpp
              swig -python -c++ -o src/pykcs11_wrap.cpp src/pykcs11.i
              :1: Error: Unable to find 'swig.swg'
              :3: Error: Unable to find 'python.swg'
              src/pykcs11.i:43: Error: Unable to find 'cdata.i'
              src/pykcs11.i:45: Error: Unable to find 'typemaps.i'
              src/pykcs11.i:46: Error: Unable to find 'std_vector.i'
              error: command 'swig' failed with exit status 1
              ----------------------------------------

The solution is to set an environment variable which indicates where swig should find its library files; in this case when SWIG is installed via the apt buildpack, the library env var would need to be:

SWIG_LIB=/home/vcap/deps/0/apt/usr/share/swig3.0

Steps to reproduce:

Push a minimal app with these files:

manifest.yml

applications:
  - buildpacks:
      - https://github.com/cloudfoundry/apt-buildpack.git
      - python_buildpack
    timeout: 180
    env:
      SWIG_LIB: /home/vcap/deps/0/apt/usr/share/swig3.0 

apt.yml

packages:
- swig

requirements.txt

pykcs11

The issue seems to be that environment variables are not available at the droplet build step. So at the point the buildpack does pip install -r requirements.txt $SWIG_LIB is not available.

Interestingly, if we remove pykcs11 from the dependencies, push the app, then ssh in and run pip install pykcs11 it installs without any errors, as the env var is available.

Please confirm where necessary:

• [x] I have included a log output
• [x] My log includes an error message
• [x] I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173561616

The labels on this github issue will be updated when the story is started.

[joshzarrabi]

@lgarvey Thanks for pointing this out, definitely looks like a bug. We will look into it.

[sophiewigmore]

After investigating this issue, we have the following context:

The SWIG_LIB environment variable is in fact available at build time, with whatever is set in the manifest.yml file. This issue is arising due to the Apt buildpack installing swig into a unique /tmp directory during the supply phase such as /tmp/contents730267330/deps/0/bin/swig. There is no way to know what that directory will be, because it is randomly generated at build-time. Because of that, we can't set the SWIG_LIB env. var to the correct path.

The reason we are able to run a manual pip install pykcs11 in the app container is because during in the final step of the build phase, the contents of that /tmp dir are moved. Nothing is in that directory during the Python buildpack supply phase (when pip install is run by the buildpack) so it fails.

[sophiewigmore]

Based on our investigation, this would require code changes to the Python buildpack. Currently, this isn't a priority for us to fix, so I'm going to close out this issue. Please free to reopen this issue if this is a critical problem for you

[lgarvey]

Is there a way to work around the issue?

On Wed, 7 Apr 2021 at 14:49, Sophie Wigmore @.***> wrote:

Based on our investigation, this would require code changes to the Python buildpack. Currently, this isn't a priority for us to fix, so I'm going to close out this issue. Please free to reopen this issue if this is a critical problem for you

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cloudfoundry/python-buildpack/issues/185#issuecomment-814929519, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB2CY3OU4UL7CKT2S4J3ZTTHRPFNANCNFSM4OLH4WDQ .

[sophiewigmore]

Hey @lgarvey, this is an unsatisfying answer but with the way the buildpack currently works, no.

The problem is that pykcs looks for swig in the predefinedSWIG_LIB location, but the apt buildpack installs swig into a "random" temp location, there's no easy workaround to define what path we want ahead of time. This is kind of a unique use case, since other packages use common conventions to lookup packages, but pykcs does not.

There are a few options here:

If you are open to contributing, we have seen an issue like this before with the RGDAL library/R buildpack/Apt buildpack. @ryanmoran wrote up this Gist document that outlines the changes that had to be made in order to add an env.sh file into source code that can modify the environment at build-time. The document corresponds with this commit in the R buildpack. Something similar would need to happen in the Python buildpack.

This process could be simpler with our newer Cloud Native Buildpacks implementations, since they install dependencies into unchanging locations, rather than random directories. This would still involve a contribution, and a switch to different buildpacks.

You could also reach out to pykcs and see if they would be open to a more standard lookup of the swig package.