Only run the openssl_certificate_loader helper once

[dmikusa]

Describe the Enhancement

In most situations, the openssl_certificate_loader helper will run once, just before the application starts. This is the assumed behavior.

The problem is that it can be triggered multiple times and each time it runs it will modify the truststore. For example, if you docker exec into the container using the launcher then it'll run this again. If you use the Paketo health checker buildpack it'll get triggered every time the health check runs (because presently it's running the health check through the launcher).

It's not clear the implications of having this run many times. We have not had reports, but this may cause unforeseen problems.

Possible Solution

Ensure that the process only runs once. I can think of a couple of ways to do this:

1. Look at the running processes. Check if there is a java process running. If so, then exit. If not, then run as normal.
2. Create a lock file in the /tmp directory the first time this process runs. If that file exists, then exit. If not, then run as normal.

You could in theory have a race condition on checking the process list or for the existence of a file, but that should be unlikely. While this exec.d helper can be triggered multiple times, I've not seen any cases where it's triggered simultaneously.

Motivation

1. Performance. Do less work.
2. Prevent unexpected/weird behavior.

[Dr4K4n]

Coming from /cnb/process/healthcheck triggers start of JVM I have a suspicion that this problem caused me even more trouble: You basically say that the trust store is rebuilt every time the health check runs. We have several @Scheduled jobs inside our Spring Application that run every minute and use a RestTemplate (newly constructed each time) to contact external webservices. Some months ago (and this might coincide with activation of the health check) we started to see the following exception at random times javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available.

We spent quite some time debugging and troubleshooting it, but we couldn't reproduce it on our local machines (where we run the Java application directly in IntelliJ - without docker)... For the time being we mitigated the problem by constructing the RestTemplate at the start of the application and caching. Unfortunately we use one library that seems to build a TrustManager by itself, this one still caused problems.

Since we switched to paketobuildpacks/builder-jammy-full and curl for the health-check the problem does no longer occur, that makes sense right? Looking forward to getting this solved, so we can switch back to the paketo health checker.

[dmikusa]

@Dr4K4n Yes, apologies. That sounds plausible. If the truststore is being recreated when you try to read it, you might see a problem like that.

[serut]

Same issue here. Our service has healthcheck activated and gets this error few times per day :

S3ClientException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available

[dmikusa]

For folks hitting this due to the health-checker buildpack, we've released health-checker 2.0.0, which no longer provides the option to start the health-check binary through the launcher. Instead, going forward you'll invoke the health check binary directly.

The output of the buildpack has changed to provide the location of those binaries, you can invoke the binary directly or the convenience symlink that we create.

  Tiny Health Checker 0.27.0: Reusing cached layer
    The Tiny Health Checker binary is available at /layers/paketo-buildpacks_health-checker/thc/bin/thc
    A symlink is available at /workspace/health-check for your convenience

Please note that since we're not using the launcher to invoke the health check binary, we cannot set env variables on the image at build time. All environment variables to configure the health check binary must be set when you run the container.

The release notes are here: https://github.com/paketo-buildpacks/health-checker/releases/tag/v2.0.0

[serut]

To make the change

Before:

    healthcheck: 
      test: ['CMD', '/cnb/process/health-check']
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 2m

After:

    healthcheck: 
      test: ['CMD', '/layers/paketo-buildpacks_health-checker/thc/bin/thc']
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 2m
    environment: 
      THC_PORT: 9093
      THC_PATH: /actuator/health

``