Failed to update CACERTS when readOnlyRootFilesystem is set

[belfo]

Expected Behavior

At startup it should create the new cacert in different folder (like tmp) that can be mounted with write rights

Current Behavior

Currently it try to modify it in place WARNING: Unable to add container CA certificates to JVM because /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts is read-only

Possible Solution

Steps to Reproduce

1. Run a builded container with readOnlyRootFilesystem set to true

Motivations

It's mandatory to use readOnlyRootFilesystem in our env

[anthonydahanne]

hello, There are 2 ways to add a ca cert to the JVM truststore: either at runtime (what you're describing) or build time.

Have you considered adding your ca cert during buildtime ?

pack build demo:1.0.0-SNAPSHOT \
  --env SERVICE_BINDING_ROOT=/platform/bindings \
  --env BP_JVM_VERSION=17 \
  --volume $PWD/binding/ca-certificates/:/platform/bindings/my-certificates \
  --builder=paketobuildpacks/builder:base

with $PWD/binding/ca-certificates/ looking like: https://github.com/anthonydahanne/java-applications-containerized/tree/master/c-buildpacks/binding/ca-certificates ?

You'll see during the build:

Paketo Buildpack for CA Certificates 3.5.1
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
  CA Certificates: Contributing to layer
    Added 1 additional CA certificate(s) to system truststore
    Writing env.build/SSL_CERT_DIR.append
    Writing env.build/SSL_CERT_DIR.delim
    Writing env.build/SSL_CERT_FILE.default

[belfo]

Hello @anthonydahanne

but this will not add it to the jvm trustore but only to the system one. At startup the jvm one is updated based on the system one. this one fail.

And in any case that would imply having 1 image per env, (as the CA is different in DEV/PROD)

Regards

[dmikusa]

Presently, the certificate loader will read and then write back to the same location.

I think we should look at modifying this so that the certificate loader can write to an alternative location. Then we could modify the helper to check if it's running in a read-only filesystem and write to /tmp instead. It would then need to set -Djavax.net.ssl.trustStore=/tmp/<modified-truststore>.

I'm not 100% fixed on that workflow though. If it's easier to have certificate loader check for a read-only filesystem that would work too. Also, by check for a read-only filesystem I really just mean check if it can write to the truststore location or possibly just attempt to and handle the error by falling back to writing to /tmp.

[patst]

We are facing the same issue in our environment. All our pods must be configured with readOnlyRootFilesystem and we have custom CAs. They are added to the system store during build time but cannot be copied to the trust store during runtime.

We can mount an emptyDir volume to pods during runtime. If it would be possible to configure the location of the modified trustStore file we could copy it to the emptyDir volumes location. Would be great if -Djavax.net.ssl.trustStore=/tmp/<modified-truststore> is set automatically, but maybe it would be possible to specify it in the JAVA_TOOL_OPTIONS.

As an alternative: Wouldn't it be possible to run the ./openssl-certificate-loader during build time? All certs are present and could be imported from the system store. The cacerts file should be writeable at the moment.

[dmikusa]

I think there are two ways we could support this:

1. Add an option similar to ca-certificates' BP_EMBED_CERTS. If this is set, then we would update the JVM's truststore at build time and that would be included in the image itself.

2. We can modify the certificate loader to an alternative location. Then we could modify the helper to check if it's running in a read-only filesystem and write to /tmp/<random-file> instead. It would then need to set -Djavax.net.ssl.trustStore=/tmp/<random-file>. We should also do a check to see if any certs actually need to be added at runtime and skip if it's not actually changing, that way you only need to have writable /tmp if you need to do this at runtime.

[belfo]

I found the option 2 more useful.

[loewenstein]

We'll likely start on a pull request for this soon.

[dmikusa]

@loewenstein - Out of curiosity, which option are you planning to implement? Or both? I think my preference would be to have both. They both seem useful, but we're thankful for any contribution.

[loewenstein]

@dmikusa Let me cite from our story :)

There are already two proposals for achieving this. Let's pick one and create a PR (or do both even - if that choice seems to make sense).

[c0d1ngm0nk3y]

Out of curiosity, which option are you planning to implement? Or both? I think my preference would be to have both. They both seem useful, but we're thankful for any contribution.

We finally start with that :) tbh, I would prefer option 2 for 3 reasons:

1. @belfo would prefer this (see comment)
2. The use case described here doesn't sound uncommon to me and baking the ca into the image seems a bit unnecessary to me.
3. Option 1 would only solve some use cases and option 2 should solve all, right?