xmlsec in python

[Lewiscowles1986]

What version of Cloud Foundry and CF CLI are you using?

api: "2.139.0" (oddly one of the fields says version: 0... troubling) cli: cf version 6.42.0+0cba12168.2019-01-10

What version of the buildpack you are using?

v1.6.36

If you were attempting to accomplish a task, what was it you were attempting to do?

Deploy an app which has deployed for > 1 year

What did you expect to happen?

App deploys

What was the actual behavior?

   2019-08-16T12:02:13.43+0100 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python3.6/site-packages/onelogin/saml2/auth.py", line 14, in <module>
   2019-08-16T12:02:13.43+0100 [APP/PROC/WEB/0] ERR     import xmlsec
   2019-08-16T12:02:13.43+0100 [APP/PROC/WEB/0] ERR ImportError: libxmlsec1-openssl.so.1: cannot open shared object file: No such file or directory

Please confirm where necessary:

• [x] I have included a log output
• [x] My log includes an error message
• [x] I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/167980606

The labels on this github issue will be updated when the story is started.

[Lewiscowles1986]

The wheel misses the shared object because it is a runtime dependency.

To fix this I used the attic multi-buildpack (not great as my understanding is that attic is deprecated)

apt.yml

---
packages:
- libxmlsec1-openssl

multi-buildpack.yml

---
  buildpacks:
  - https://github.com/cloudfoundry/apt-buildpack#v0.1.8
  - https://github.com/cloudfoundry/python-buildpack#v1.6.36

This seems to work and should have legs for the future. I'm a little upset it came to this.

[zmackie]

Not sure what the issue is here. We don't include every possible os package on our rootfs and the use of the apt buildpack to install packages is a perfectly normal way to install packages. Multi-buildpack support is also fully supported (you could that multi-buildpack.yml content into your main manifest.yml per https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#buildpack)

[Lewiscowles1986]

Multi-buildpack in manifest.yml

Building the buildpacks into the manifest.yml brought up errors. It didn't work, but was one of the first things attempted.

Something about an API version. We tried pulling from a newer cloudfoundry deploy docker image, I personally went through the docs. We received many other errors, but were unable to deploy using that tip.

Not including the kitchen sink in a base image

We don't include every possible os package on our rootfs

Nobody asked you to and that was totally unnecessary misrepresentation of what this issue calls out.

This shared object used to be included when deployed on cflinuxfs3 using the python buildpack. AFAIK the buildpack is our only way to modify the application-specific system dependencies.

Work done prior

Up until this point, I've isolated and deployed changes including vendored dependencies from an earlier successful build. I can prove none of our code built that shared object and deploys worked (the dependency is very old).

Multiple engineers poured over the CF docs including the multi-buildpack docs you linked. None of it worked or signposted deeper troubleshooting docs. To avoid reading source-code of the managed system we pay for to avoid the burden of managing our own platform, we took an inventory.

1. We don't go to the python package manager because we vendor dependencies.
2. Prior buildpacks seem to also not work (we pinned after successful remittance and on several attempts to remit anyway)
3. None of the uploaded artifacts from the past 6 months contain this shared object.
4. We only know something changed between June/July and August, which happens to have come at a time when this specific product codebase needing this shared object was sunset and was receiving no updates.
5. Our last successful deploy commit hash + vendored dependencies with prior buildpack would not deploy either.

As this is outside of our codebase and we only use official buildpacks, we looked into several other ways to remit and eliminated everything besides this buildpack including manually including the shared object in a python wheel.

I came to the place those system-level dependencies are supplied as I believe is reasonable to raise an issue so that others encountering this on this platform would have one place to look for solutions.