Unable to write cacerts on read-only file system with Java 21

[AndersClausen]

After upgrading a simple Spring Boot service (v. 3.2.1) to use Java 21, we experience an error at deployment time that didn't occur when using Java 17 (Liberica distribution).

Expected Behavior

The service should start up as normal after deployment.

Current Behavior

After upgrading a simple Spring Boot service (v. 3.2.1) to use Java 21, we experience the following error at deployment time:

Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx1686770K -XX:MaxMetaspaceSize=103181K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 2G, Thread Count: 50, Loaded Class Count: 15803, Headroom: 0%)
Enabling Java Native Memory Tracking
Using readonly truststore: /tmp/truststore
Adding 137 container CA certificates to JVM truststore
unable to load certificates
unable to write keystore
open /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts: read-only file system
ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader': exit status 1

The OCI image was created using spring-boot-maven-plugin with the following settings:

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
            </plugin>

and running the mvn spring-boot:build-image command. The service does not have any custom configuration with certificates and worked perfectly well with Java 17. It's deployed to Kubernetes and the OCI image is running in a read-only file system mode specified by setting the readOnlyRootFilesystem property of the Kubernetes SecurityContext to true. This is something our Cyber Security team has advised us to do and worked with Java 17.

Possible Solution

Not sure but puzzled to why it's trying to write certificates to the filesystem using Java 21 when that didn't happen with Java 17.

Steps to Reproduce

1. Create a Spring Boot service with version 3.2.1 and Java 21 Liberica distribution.
2. Create an OCI image with spring-boot-maven-plugin (configuration and command as above).
3. Deploy to Kubernetes where the readOnlyRootFilesystem property of the SecurityContext is set like this:

   securityContext:
   runAsRoot: true
   readOnlyRootFilesystem: true

Motivations

In order to make the deployment successful, we have to set the readOnlyRootFilesystem to false which means we're less secure and goes against our company's cyber security recommendations.

[dmikusa]

This may be related to https://github.com/paketo-buildpacks/libjvm/pull/344.

It could also be related to a recent change in the underlying certificate loading to support Java 18+. Previously, we just did not load CA certificates if you were on Java 18+. This is because the keystore format changed and we had trouble with the new format and libraries being used. Those have all been resolved, and so it should be importing system CA certs into the JVM's default trust store now. This is the behavior that allows you to add your own trusted CA certs to your container (see the ca-certificates buildpack). It is possible that previously it was just skipping this since we couldn't write to the keystore, and now it's writing and so you're seeing the error.

This ability to import ca certs should work so long as you have a writable /tmp filesystem. You can make everything else read-only, but we need at least a writable /tmp. I was also under the impression that it would just give you a warning if it couldn't write, not fail, so it sounds like there could be a regression.

Thanks for this report.

[anthonydahanne]

Also related: https://github.com/paketo-buildpacks/libjvm/pull/359

[AndersClausen]

Hi all. Just wondering if there's an update on this issue. Cheers

[dmikusa]

I haven't had time to dig and try reproducing it myself, but looking at the error and tracing it through the code. I think this is the spot that's triggering the error.

https://github.com/paketo-buildpacks/libjvm/blob/main/keystore.go#L168

Oddly, the previous lines check for write access and must succeed, otherwise it would return and not trigger the error.

    if unix.Access(k.location, unix.W_OK) != nil {
        return nil
    }

I suspect that we need to handle this differently because it's a read-only file system, not a write access issue. Again, I haven't had the time to reproduce to investigate possible fixes.

@c0d1ngm0nk3y or @modulo11 I think you've been in this part of the code base. Have you seen any issues like this? Thanks

[modulo11]

I can reproduce the issue. @AndersClausen you need to provide a writable TMP folder (emptydir is sufficient).

[dmikusa]

@modulo11 I thought this was supposed to skip, even if there is no writable temp dir. This PR is even adding warnings to indicate when that happens.

[modulo11]

@modulo11 I thought this was supposed to skip, even if there is no writable temp dir. This PR is even adding warnings to indicate when that happens.

I did some additional tests and it's working properly (without any writable folders) with the latest release of libjvm or Bellsoft etc. However, the latest builder does include a version of the Java buildpack which does not include the fix.

[dmikusa]

@AndersClausen I apologize. There was a problem with the Paketo builder distribution and it had not picked up the latest buildpacks. This has been fixed. If you make sure to pull the latest builder, are you still seeing issues with it?

[AndersClausen]

Hi @dmikusa - thanks for the update. I'll try and get it tested within the next few days and get back to you with an update.