search

paketo-buildpacks / ca-certificates

A Cloud Native Buildpack that adds custom CA certificates to a build and a created image
Apache License 2.0
24 stars 10 forks source link

Adding ca-certificates in runtime does not work #237

Open paulc0te opened 1 month ago

paulc0te commented 1 month ago

Expected Behavior

Adding certificates at runtime must not break work of certificates added during build.

Current Behavior

The certificates that I added during build stop working after I add the certificate at runtime.

Possible Solution

If I add the same certificates only during build, then everything works

Steps to Reproduce

I'm building a spring boot application image using a gradle task that runs paketo-buildpacks. After build, the application works in k8s.

Directory with certificates when building an image

bindings/ca-certificates  
cert1.pem  
...  
cert12.pem  
type <- (ca-certificates)

gradle task with which the application image is built

bootBuildImage {
    builder = "${dockerRepoUrl}/paketobuildpacks/builder:0.3.158-base"
    runImage = "${dockerRepoUrl}/paketobuildpacks/run:1.2.49-base-cnb"
    bindings = [
            "${project.rootDir}/bindings/ca-certificates:/platform/bindings/ca-certificates",
            "${project.rootDir}/bindings/depmapping:/platform/bindings/depmapping"
    ]
}

The command I run the build with

gradle clean bootBuildImage

Part of the Buildlog that I see when building

...
[creator]     Paketo Buildpack for CA Certificates 3.5.1
[creator]       https://github.com/paketo-buildpacks/ca-certificates
[creator]       Launch Helper: Contributing to layer
[creator]         Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[creator]       CA Certificates: Contributing to layer
[creator]         Added 12 additional CA certificate(s) to system truststore
[creator]         Writing env.build/SSL_CERT_DIR.append
[creator]         Writing env.build/SSL_CERT_DIR.delim
[creator]         Writing env.build/SSL_CERT_FILE.default
...

Attached is the full build_log

The env variable which points to the directory in which the ca-certificates directory is located

SERVICE_BINDING_ROOT: /paketo-certs/platform/bindings

Directory with certificates in a container

ls /paketo-certs/platform/bindings/ca-certificates
cert13.pem
type <- (ca-certificates)

Application start log

Added 1 additional CA certificate(s) to system truststore
Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345042K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58504, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 125 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
2024-07-10T17:01:26+03:00   Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345042K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
2024-07-10T17:01:36+03:00   2024-07-10 17:01:36.380 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :Starting AtsArmApplication using Java 11.0.18 on arm-bb9d9646-msvkx with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
...

After the application starts, I try to use certificates, but they don't work. I am getting the following error

[BackgroundTask-0-psz-admin] [MDC : jmixUser=psz-admin] c.a.minio.filestorage.MinioFileStorage  :Error saving file to minio storage
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

_JAVAHOME points to the correct cacerts location

JAVA_HOME=/layers/paketo-buildpacks_bellsoft-liberica/jre

ls /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security
blocked.certs  cacerts  default.policy  public_suffix_list.dat

All certificates are recorded in cacerts and are working. I checked this by exporting and using curl.

I noticed that the certificates added in cacerts during build look like this

/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:C1

Certificates added at runtime look like this

/tmp/ca-certificates1702841895/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:32:D4:D8

I tried updating paketo-buildpacks/ca-certificates to the latest version but it did not solve the problem

Motivations

I need to use 2 of these methods - because during the build phase paketo-buildpacks accesses the https repository for its dependencies. Accordingly, he should already have a certificate. What is the reason for this behavior of paketo-buildpacks/ca-certificates ?

dmikusa commented 1 month ago

The certificates that I added during build stop working after I add the certificate at runtime.

Are you embedding the build-time certs into the image? I don't see that in your build output, but I wanted to double check.

To do that you would set BP_EMBED_CERTS=true. By default, it doesn't to embed the ca certificates from build into the container so you'd need to also include them with your list of runtime ca certificates.

I'm still not 100% sure that will work, but that would be the first thing I'd try.

paulc0te commented 1 month ago

@dmikusa, thanks for your advice, the certificates actually made it into the container and are now here

ls /layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/
569e89cc.0  66990ff1.0  720381b8.0  8b90f171.0  8b90f171.1  8b90f171.2  921c95d1.0  a316e707.0  a316e707.1  a316e707.2  a316e707.3  a316e707.4

But this does not solve my problem, I need the java application to work after adding certificates. Now I'm getting the same error

[BackgroundTask-1-psz-admin] [MDC : jmixUser=psz-admin] c.a.minio.filestorage.MinioFileStorage  :Error saving file to minio storage
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This is part of the application start log

Added 1 additional CA certificate(s) to system truststore
Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58521, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

 :: Spring Boot ::                (v2.7.4)

2024-07-11 17:06:45.991 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :Starting AtsArmApplication using Java 11.0.23 on arm-c8cbc5d76-kl2pb with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-07-11 17:06:45.996 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :No active profile set, falling back to 1 default profile: "default"
2024-07-11 17:06:54.759 -  INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Bootstrapping Spring Data Jmix repositories in DEFAULT mode.
2024-07-11 17:06:55.271 -  INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Finished Spring Data repository scanning in 502 ms. Found 1 Jmix repository interfaces.

This is part of the build log

:bootBuildImage
   > Pulling builder image 'site.com/paketobuildpacks/builder:0.3.158-base' ..................................................
   > Pulled builder image 'site.com/paketobuildpacks/builder@sha256:baf8e13d21e274361a59557f717969f15794271e212145894bbff46a4173dc08'
   > Pulling run image 'site.com/paketobuildpacks/run:1.2.49-base-cnb' ..................................................
   > Pulled run image 'site.com/paketobuildpacks/run@sha256:3fbc7987519710abebafbfa67827b226a4b88630678c149a2dfb494cb73bd150'
   > Pulling buildpack image 'site.com/paketobuildpacks/java:latest' ..................................................
   > Pulled buildpack image 'site.com/paketobuildpacks/java@sha256:9f292e740e044c7c364bf5d7e666908f8a1860800d216212da7cd15e9e921bad'
   > Executing lifecycle version v0.15.3
   > Using build cache volume 'pack-cache-d871107f3bf9.build'

   > Running creator
      [creator]     ===> ANALYZING
      [creator]     Previous image with name "site.com/ats/arm-base:D221" not found
      [creator]     ===> DETECTING
      [creator]     6 of 26 buildpacks participating
      [creator]     paketo-buildpacks/ca-certificates   3.8.2
      [creator]     paketo-buildpacks/bellsoft-liberica 10.8.1
      [creator]     paketo-buildpacks/syft              1.47.1
      [creator]     paketo-buildpacks/executable-jar    6.10.1
      [creator]     paketo-buildpacks/dist-zip          5.8.1
      [creator]     paketo-buildpacks/spring-boot       5.30.1
      [creator]     ===> RESTORING
      [creator]     ===> BUILDING
      ..
      [creator]     Paketo Buildpack for CA Certificates 3.8.2
      [creator]       https://github.com/paketo-buildpacks/ca-certificates
      [creator]       Build Configuration:
      [creator]         $BP_EMBED_CERTS                    true   Embed certificates into the image
      [creator]         $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
      [creator]         $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
      ..
      [creator]       CA Certificates: Contributing to layer
      [creator]         Embedding CA certificate(s)
      [creator]         Added 12 additional CA certificate(s) to system truststore

Full build log here

Here I am retrieving all the added certs from the java truststore. It can be seen that all 13 certificates are in place

KEYTOOL=/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/keytool
CACERTS=/layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
$KEYTOOL -list \
        -storepass $PASS \
        -keystore $CACERTS
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
/tmp/ca-certificates1300581086/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11

I also increased the JRE version to 11.0.23 and paketo-buildpacks/ca-certificates to 3.8.2

I would be grateful for any advice on what else I can try.

dmikusa commented 1 month ago

From everything you've show here, it all appears to be working. We can see the CA certs being added to the image, we can see the exec.d helper loading your additional ca cert at runtime. We can see the exec.d helper for the JVM running and pulling in 13 additional certificates to the JVM's truststore. That's everything it should be doing, and the certs should be accessible to your Java app at this point.

If it's still not trusting the connection, then you'd need to do some more investigation of the specific connection to see why it can't match up the certs, there could be a cert missing still. The advise here is helpful for doing that -> https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html

If you see something that looks off from what the buildpack is doing, let me know. Happy to look again, but I'm not seeing anything that looks incorrect with what you've posted here.

paulc0te commented 1 month ago

The certificates are indeed written to the java truststore, but the problem is that they stop working as soon as I add another one using the at runtime method. If I add all the same certificates, but during build, everything works. The certificates are all checked and working, the application is working.

In this regard, I have several questions: 1) Can these two methods of adding certificates work together (At build time, At runtime)? 2) I have a suspicion that helper, in At runtime mode, adds certificates either after the application starts loading or in parallel with it, which leads to the fact that some or all certificates become non-working. Since after adding to the java truststore, the application must be restarted. Can you refute or confirm this?

Thank you very much for the article, I will take a look at it.

dmikusa commented 1 month ago

Ok, thanks for expanding on that.

Can these two methods of adding certificates work together (At build time, At runtime)?

That sounds like the million-dollar question here. I'm not totally sure off-hand, but I'll take a look.

I have a suspicion that helper, in At runtime mode, adds certificates either after the application starts loading or in parallel with it, which leads to the fact that some or all certificates become non-working.

All of the exec.d helpers run before your app starts, so there's no need to restart the app. All of the certs are loaded before your app starts. It's weird because your cert does have all of the certs loaded both build + runtime when you list the output of the keystore file. That would leave me to believe it's doing the right thing.

Could you send me the same output for the keystore when you put all of the ca certs in at build time and your app works so I can compare them?

Thanks

paulc0te commented 1 month ago

I disabled SERVICE_BINDING_ROOT. I also had to make BP_EMBED_CERTS: false. Without this, my application still didn't work even when adding certificates during build. This is news to me.

Next, I built the application image using the mode of adding certificates during build. And it still works after this build. But I really want to configure all this in runtime mode.

Here is the application start log

Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58521, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

 :: Spring Boot ::                (v2.7.4)

2024-07-12 00:24:55.272 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :Starting AtsArmApplication using Java 11.0.23 on arm-bf8b68589-56lnr with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-07-12 00:24:55.277 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :No active profile set, falling back to 1 default profile: "default"
2024-07-12 00:25:04.777 -  INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Bootstrapping Spring Data Jmix repositories in DEFAULT mode.

Here is part of the buildlog

> Task :bootBuildImage
:bootBuildImage
  Building image 'site.com/ats/arm-base:D223'

   > Pulling builder image 'site.com/paketobuildpacks/builder:0.3.158-base' ..................................................
   > Pulled builder image 'site.com/paketobuildpacks/builder@sha256:baf8e13d21e274361a59557f717969f15794271e212145894bbff46a4173dc08'
   > Pulling run image 'site.com/paketobuildpacks/run:1.2.49-base-cnb' ..................................................
   > Pulled run image 'site.com/paketobuildpacks/run@sha256:3fbc7987519710abebafbfa67827b226a4b88630678c149a2dfb494cb73bd150'
   > Pulling buildpack image 'site.com/paketobuildpacks/java:latest' ..................................................
   > Pulled buildpack image 'site.com/paketobuildpacks/java@sha256:9f292e740e044c7c364bf5d7e666908f8a1860800d216212da7cd15e9e921bad'
   > Executing lifecycle version v0.15.3
   > Using build cache volume 'pack-cache-6e2b2907d17b.build'

   > Running creator
      [creator]     ===> ANALYZING
      [creator]     Previous image with name "site.com/ats/arm-base:D223" not found
      [creator]     ===> DETECTING
      [creator]     6 of 26 buildpacks participating
      [creator]     paketo-buildpacks/ca-certificates   3.8.2
      [creator]     paketo-buildpacks/bellsoft-liberica 10.8.1
      [creator]     paketo-buildpacks/syft              1.47.1
      [creator]     paketo-buildpacks/executable-jar    6.10.1
      [creator]     paketo-buildpacks/dist-zip          5.8.1
      [creator]     paketo-buildpacks/spring-boot       5.30.1
      [creator]     ===> RESTORING
      [creator]     ===> BUILDING
...
      [creator]     Paketo Buildpack for CA Certificates 3.8.2
      [creator]       https://github.com/paketo-buildpacks/ca-certificates
      [creator]       Build Configuration:
      [creator]         $BP_EMBED_CERTS                    false  Embed certificates into the image
      [creator]         $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
      [creator]         $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
...
      [creator]       CA Certificates: Contributing to layer
      [creator]         Added 13 additional CA certificate(s) to system truststore
...
      [creator]     Paketo Buildpack for BellSoft Liberica 10.8.1
      [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
      [creator]       Build Configuration:
      [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
      [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
      [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
      [creator]         $BP_JVM_VERSION              11.*                                                         the Java version
      [creator]       Launch Configuration:
      [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
      [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
      [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
      [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
      [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
      [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
      [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
      [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
      [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
      [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
      [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
      [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
      [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
      [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
      [creator]         Using Java version 11.* from BP_JVM_VERSION
...
      [creator]       BellSoft Liberica JRE 11.0.23: Contributing to layer
      [creator]         Downloading from https://site.com/repository/elk-raw/bell-sw/Liberica/releases/download/11.0.23+12/bellsoft-jre11.0.23+12-linux-amd64.tar.gz
      [creator]         Verifying checksum
      [creator]     Writing Syft SBOM at /layers/paketo-buildpacks_bellsoft-liberica/jre.sbom.syft.json: {Artifacts:[{ID:a640c2f785dc95aa Name:BellSoft Liberica JRE Version:11.0.23 Type:UnknownPackage FoundBy:libpak Locations:[{Path:buildpack.toml}] Licenses:[GPL-2.0 WITH Classpath-exception-2.0] Language: CPEs:[cpe:2.3:a:oracle:jre:11.0.23:*:*:*:*:*:*:*] PURL:pkg:generic/bellsoft-jre@11.0.23?arch=amd64}] Source:{Type:directory Target:/layers/paketo-buildpacks_bellsoft-liberica/jre} Descriptor:{Name:syft Version:0.32.0} Schema:{Version:1.1.0 URL:https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json}}
      [creator]         Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
      [creator]         Adding 137 container CA certificates to JVM truststore
...
      [creator]     Paketo Buildpack for Spring Boot 5.30.1
      [creator]       https://github.com/paketo-buildpacks/spring-boot
      [creator]       Build Configuration:
      [creator]         $BPL_JVM_CDS_ENABLED                 false  whether to enable CDS optimizations at runtime
      [creator]         $BPL_SPRING_AOT_ENABLED              false  whether to enable Spring AOT at runtime
      [creator]         $BP_JVM_CDS_ENABLED                  false  whether to enable CDS & perform JVM training run
      [creator]         $BP_SPRING_AOT_ENABLED               false  whether to enable Spring AOT
      [creator]         $BP_SPRING_CLOUD_BINDINGS_DISABLED   false  whether to contribute Spring Boot cloud bindings support
      [creator]         $BP_SPRING_CLOUD_BINDINGS_VERSION    1      default version of Spring Cloud Bindings library to contribute
      [creator]       Launch Configuration:
      [creator]         $BPL_SPRING_CLOUD_BINDINGS_DISABLED  false  whether to auto-configure Spring Boot environment properties from bindings
      [creator]         $BPL_SPRING_CLOUD_BINDINGS_ENABLED   true   Deprecated - whether to auto-configure Spring Boot environment properties from bindings
      [creator]       Creating slices from layers index
      [creator]         dependencies (296.8 MB)
      [creator]         spring-boot-loader (258.2 KB)
      [creator]         snapshot-dependencies (0.0 B)
      [creator]         application (22.5 MB)

Here is the full buildlog in debug mode

This is the part of the trust store in which you can see all 13 added certificates during build. I replaced the ending with 11. I don’t know why))

/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
paulc0te commented 1 month ago

For testing, I temporarily changed my infrastructure and now I don’t need ca-certificates when building the application. Now I am adding the same certificates but only at runtime. However, my application is still not working.

Just in case, I’ll clarify that my application is spring boot running in kubernetes. The directory with ca-certificates is copied inside the container via mount k8s secrets. There is also an env variable (SERVICE_BINDING_ROOT=/paketo-certs/platform/bindings) that points to the directory with ca-certificates. Among the certificates there are chains of certificates. Each part of the chain is divided into PEM files.

Here is the application start log

Added 13 additional CA certificate(s) to system truststore
Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58521, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

 :: Spring Boot ::                (v2.7.4)

2024-07-12 14:55:04.956 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :Starting AtsArmApplication using Java 11.0.23 on arm-c8cbc5d76-5gjw9 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-07-12 14:55:04.969 -  INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication           :No active profile set, falling back to 1 default profile: "default"
2024-07-12 14:55:19.148 -  INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Bootstrapping Spring Data Jmix repositories in DEFAULT mode.

Here is part of the buildlog

:bootBuildImage
  Building image 'site.com/ats/arm-base:D224'

   > Pulling builder image 'site.com/paketobuildpacks/builder:0.3.158-base' ..................................................
   > Pulled builder image 'site.com/paketobuildpacks/builder@sha256:baf8e13d21e274361a59557f717969f15794271e212145894bbff46a4173dc08'
   > Pulling run image 'site.com/paketobuildpacks/run:1.2.49-base-cnb' ..................................................
   > Pulled run image 'site.com/paketobuildpacks/run@sha256:3fbc7987519710abebafbfa67827b226a4b88630678c149a2dfb494cb73bd150'
   > Pulling buildpack image 'site.com/paketobuildpacks/java:latest' ..................................................
   > Pulled buildpack image 'site.com/paketobuildpacks/java@sha256:9f292e740e044c7c364bf5d7e666908f8a1860800d216212da7cd15e9e921bad'
   > Executing lifecycle version v0.15.3
   > Using build cache volume 'pack-cache-961e92e2c9d2.build'

   > Running creator
      [creator]     ===> ANALYZING
      [creator]     Previous image with name "site.com/ats/arm-base:D224" not found
      [creator]     ===> DETECTING
      [creator]     6 of 26 buildpacks participating
      [creator]     paketo-buildpacks/ca-certificates   3.8.2
      [creator]     paketo-buildpacks/bellsoft-liberica 10.8.1
      [creator]     paketo-buildpacks/syft              1.47.1
      [creator]     paketo-buildpacks/executable-jar    6.10.1
      [creator]     paketo-buildpacks/dist-zip          5.8.1
      [creator]     paketo-buildpacks/spring-boot       5.30.1
      [creator]     ===> RESTORING
      [creator]     ===> BUILDING
      [creator]     Paketo Buildpack for CA Certificates 3.8.2
      [creator]       https://github.com/paketo-buildpacks/ca-certificates
      [creator]       Build Configuration:
      [creator]         $BP_EMBED_CERTS                    false  Embed certificates into the image
      [creator]         $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
      [creator]         $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
      [creator]     Paketo Buildpack for BellSoft Liberica 10.8.1
      [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
      [creator]       Build Configuration:
      [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
      [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
      [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
      [creator]         $BP_JVM_VERSION              11.*                                                         the Java version
      [creator]       Launch Configuration:
      [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
      [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
      [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
      [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
      [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
      [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
      [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
      [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
      [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
      [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
      [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
      [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
      [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
      [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
      [creator]         Using Java version 11.* from BP_JVM_VERSION
      [creator]       BellSoft Liberica JRE 11.0.23: Contributing to layer
      [creator]         Downloading from http://site.com:8081/repository/elk-raw/bell-sw/Liberica/releases/download/11.0.23+12/bellsoft-jre11.0.23+12-linux-amd64.tar.gz
      [creator]         Verifying checksum
      [creator]     Writing Syft SBOM at /layers/paketo-buildpacks_bellsoft-liberica/jre.sbom.syft.json: {Artifacts:[{ID:a640c2f785dc95aa Name:BellSoft Liberica JRE Version:11.0.23 Type:UnknownPackage FoundBy:libpak Locations:[{Path:buildpack.toml}] Licenses:[GPL-2.0 WITH Classpath-exception-2.0] Language: CPEs:[cpe:2.3:a:oracle:jre:11.0.23:*:*:*:*:*:*:*] PURL:pkg:generic/bellsoft-jre@11.0.23?arch=amd64}] Source:{Type:directory Target:/layers/paketo-buildpacks_bellsoft-liberica/jre} Descriptor:{Name:syft Version:0.32.0} Schema:{Version:1.1.0 URL:https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json}}
      [creator]         Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
      [creator]         Adding 124 container CA certificates to JVM truststore
      [creator]     Paketo Buildpack for Spring Boot 5.30.1
      [creator]       https://github.com/paketo-buildpacks/spring-boot
      [creator]       Build Configuration:
      [creator]         $BPL_JVM_CDS_ENABLED                 false  whether to enable CDS optimizations at runtime
      [creator]         $BPL_SPRING_AOT_ENABLED              false  whether to enable Spring AOT at runtime
      [creator]         $BP_JVM_CDS_ENABLED                  false  whether to enable CDS & perform JVM training run
      [creator]         $BP_SPRING_AOT_ENABLED               false  whether to enable Spring AOT
      [creator]         $BP_SPRING_CLOUD_BINDINGS_DISABLED   false  whether to contribute Spring Boot cloud bindings support
      [creator]         $BP_SPRING_CLOUD_BINDINGS_VERSION    1      default version of Spring Cloud Bindings library to contribute
      [creator]       Launch Configuration:
      [creator]         $BPL_SPRING_CLOUD_BINDINGS_DISABLED  false  whether to auto-configure Spring Boot environment properties from bindings
      [creator]         $BPL_SPRING_CLOUD_BINDINGS_ENABLED   true   Deprecated - whether to auto-configure Spring Boot environment properties from bindings

Here is the full buildlog in debug mode

This is the part of the trust store in which you can see all 13 added certificates at runtime

...
/tmp/ca-certificates2357141951/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
/tmp/ca-certificates2357141951/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
/tmp/ca-certificates2357141951/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
/tmp/ca-certificates2357141951/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
/tmp/ca-certificates2357141951/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
/tmp/ca-certificates2357141951/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
/tmp/ca-certificates2357141951/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
/tmp/ca-certificates2357141951/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
/tmp/ca-certificates2357141951/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
/tmp/ca-certificates2357141951/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
/tmp/ca-certificates2357141951/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
/tmp/ca-certificates2357141951/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
/tmp/ca-certificates2357141951/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
...

Any ideas why this is happening?

dmikusa commented 1 month ago

Added 13 additional CA certificate(s) to system truststore Adding 137 container CA certificates to JVM truststore

Well, I don't see any change to the number of certificates being loaded. It seems the same as when you had some being loaded at build and some at runtime. That's what I would expect to see.

I also did a comparison of the keystore output from your build and from your runtime and they look the same to me. The only difference is in the path from which it imported the original certs. That is to be expected and that's not going to matter. The certificate fingerprints are all the same and in the same order, that's the crucial part.

The trouble is that neither of those examples actually resulted in a working app. What I'd suggest doing is comparing the state of the keystore when it's not working with one that is working. It's likely that something is missing or different and if we can figure out what that is then we can figure out why it's missing and how to fix that.

So if you keytool list the truststore that's generated in a container where your app works and then do the same thing in the container where it doesn't work, take the outputs and diff -u them (or use your diff util of choice). It's very likely that something will be different.

paulc0te commented 1 month ago

Below is a comparison 2 files don't work and working. The only difference is in aliases. There are the same number of certificates, 295, they are added in the same order and they have the same fingerprint.

Certificate information was retrieved using the command

KEYTOOL=/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/keytool
CACERTS=/layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts

$KEYTOOL -list \
        -storepass password \
        -keystore $CACERTS
diff -u cacerts_didnot_work.txt cacerts_worked.txt

--- cacerts_didnot_work.txt     2024-07-12 09:32:18.000000000 -0400
+++ cacerts_worked.txt  2024-07-12 10:35:37.000000000 -0400
@@ -253,31 +253,31 @@
 Certificate fingerprint (SHA-256): 2C:E1:CB:0B:F9:D2:F9:E1:02:99:3F:BE:21:51:52:C3:B2:DD:0C:AB:DE:1C:68:E5:31:9B:83:91:54:DB:B7:11
 /etc/ssl/certs/ca-certificates.crt-99, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 56:8D:69:05:A2:C8:87:08:A4:B3:02:51:90:ED:CF:ED:B1:97:4A:60:6A:13:C6:E5:29:0F:CB:2A:E6:3E:DA:11
-/tmp/ca-certificates2357141951/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
-/tmp/ca-certificates2357141951/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
-/tmp/ca-certificates2357141951/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
-/tmp/ca-certificates2357141951/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
-/tmp/ca-certificates2357141951/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
-/tmp/ca-certificates2357141951/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
-/tmp/ca-certificates2357141951/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
-/tmp/ca-certificates2357141951/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
-/tmp/ca-certificates2357141951/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
-/tmp/ca-certificates2357141951/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
-/tmp/ca-certificates2357141951/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
-/tmp/ca-certificates2357141951/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
-/tmp/ca-certificates2357141951/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
 Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
 actalisauthenticationrootca [jdk], Sep 22, 2011, trustedCertEntry,
 Certificate fingerprint (SHA-256): 55:92:60:84:EC:96:3A:64:B9:6E:2A:BE:01:CE:0B:A8:6A:64:FB:FE:BC:C7:AA:B5:AF:C1:55:B3:7F:D7:60:11
dmikusa commented 1 month ago

If they're all the same, that sounds to me like the buildpack is doing the right thing, or at least it's behaving consistently in both scenarios. Ultimately, its job is to take the ca-certs from your bindings, add them to the system store (i.e. make OpenSSL see them), and then add them to the JVM's default keystore, which makes the JVM see them. In both cases, we can see them in the JVM's truststore, the ultimate destination, so it appears to be doing its job.

Beyond that, we'd have to know specifically why it's failing to trust the certificate. If you enable the TLS debugging features in the JVM, you can usually get a pretty good idea of why it's failing.

https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html

If it's failing because of something the buildpack is doing, we can certainly address that, but without knowing why the JVM is rejecting it, it's hard for me to comment further.

paulc0te commented 1 month ago

enabled debug for the application. This is all that is visible in him at the moment of the error

javax.net.ssl|DEBUG|45 E3|BackgroundTask-0-psz-admin|2024-07-12 20:23:17.707 MSK|null:-1|Consuming server Certificate handshake message (
"Certificate": {
  "certificate_request_context": "",
  "certificate_list": [  
  {
    "certificate" : {
      "version"            : "v3",
      "serial number"      : "5D 60 2C CB D4 E8 07 B1 06 9B 82 A4 77 D1 D6 1B 0F 56 E6 11",
      "signature algorithm": "SHA256withRSA",
      "issuer"             : "CN=ElkSubCA",
      "not before"         : "2022-12-21 15:53:43.000 MSK",
      "not  after"         : "2024-12-20 15:53:42.000 MSK",
      "subject"            : "CN=site.com",
      "subject public key" : "RSA",
      "extensions"         : [
        {
          ObjectId: 2.5.29.35 Criticality=false
          AuthorityKeyIdentifier [
          KeyIdentifier [
          0000: 25 74 EE 7A 70 70 CB E3   6E 36 C1 AC FE E8 D5 15  %t.zpp..n6......
          0010: 43 A9 BA C6                                        C...
          ]
          ]
        },
        {
          ObjectId: 2.5.29.19 Criticality=true
          BasicConstraints:[
            CA:false
            PathLen: undefined
          ]
        },
        {
          ObjectId: 2.5.29.37 Criticality=false
          ExtendedKeyUsages [
            serverAuth
          ]
        },
        {
          ObjectId: 2.5.29.15 Criticality=true
          KeyUsage [
            DigitalSignature
            Key_Encipherment
          ]
        },
        {
          ObjectId: 2.5.29.17 Criticality=false
          SubjectAlternativeName [
            DNSName: site.com
            DNSName: site.com00.test.elk.loc
            DNSName: site.com01.test.elk.loc
            DNSName: site.com02.test.elk.loc
            DNSName: site.com03.test.elk.loc
            IPAddress: 192.168.1.33
            IPAddress: 192.168.1.34
            IPAddress: 192.168.1.35
            IPAddress: 192.168.1.36
            IPAddress: 192.168.1.37
          ]
        },
        {
          ObjectId: 2.5.29.14 Criticality=false
          SubjectKeyIdentifier [
          KeyIdentifier [
          0000: 9F 57 13 8C B6 49 3E 65   65 3E A0 FD FB 45 65 DC  .W...I>ee>...Ee.
          0010: 9D 60 06 43                                        .`.C
          ]
          ]
        }
      ]}
    "extensions": {
      <no extension>
    }
  },
]
}
)
javax.net.ssl|DEBUG|45 E3|BackgroundTask-0-psz-admin|2024-07-12 20:23:17.707 MSK|null:-1|Ignore unavailable extension: status_request
javax.net.ssl|ERROR|45 E3|BackgroundTask-0-psz-admin|2024-07-12 20:23:17.710 MSK|null:-1|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (
"throwable" : {
  sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
dmikusa commented 1 month ago

The PKIX path error is saying that it can't find a path from the server's cert to a trusted cert in the JVMs truststore. You can walk that path on your own if you start with the issuer of the server's cert, which is "issuer": "CN=ElkSubCA",. Then look for a cert in the truststore where the subject is CN=ElkSubCA. In a nutshell, it's saying it can't find one.

If that doesn't exist in the truststore, check your bindings and confirm you have it in your bindings. If it's in your bindings, but not in the truststore, then we can look at the bindings to see why it might be skipped.

paulc0te commented 1 month ago

All necessary certificates are available in the java trust store. The application works when I add them to the image at the build stage. The application does not work when I add the same certificates in at runtime mode.

The certificate chain for a host with which SSL does not work looks like this

ATSRootCA
    ElkSubCA
        site.com      <-- host

Look, all these certificates are in the java truststore

The command with which I retrieve information about certificates

KEYTOOL=/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/keytool
CACERTS=/layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
$KEYTOOL -list \
        -v \
        -storepass $PASS \
        -keystore $CACERTS \
        -alias $CERT_ALIASE 
Alias name: /tmp/ca-certificates8216640/66990ff1.0-0
Creation date: Jan 1, 1980
Entry type: trustedCertEntry

Owner: CN=ATSRootCA
Issuer: CN=ATSRootCA
Serial number: 572e997043ba54b649c85b8aefdc3211
Valid from: Wed Jan 21 15:27:47 MSK 2009 until: Sun Jan 21 15:38:47 MSK 2029
Certificate fingerprints:
         SHA1: 7D:7F:07:9F:E5:70:0D:FB:C0:F0:49:50:F7:EC:84:E0:D3:D4:55:06
         SHA256: 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
...
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
Alias name: /tmp/ca-certificates8216640/720381b8.0-0
Creation date: Jan 1, 1980
Entry type: trustedCertEntry

Owner: CN=ElkSubCA
Issuer: CN=ATSRootCA
Serial number: 47000000366403e8395a9925a3000200000036
Valid from: Wed Dec 14 22:26:39 MSK 2022 until: Sun Jan 21 15:38:47 MSK 2029
Certificate fingerprints:
         SHA1: 32:F3:84:3A:DA:77:81:96:0A:A3:47:4F:8C:C3:35:52:04:17:67:57
         SHA256: 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
...
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
Alias name: /tmp/ca-certificates8216640/436e719d.0-0
Creation date: Jan 1, 1980
Entry type: trustedCertEntry

Owner: CN=site.com
Issuer: CN=ElkSubCA
Serial number: 5d602ccbd4e807b1069b82a477d1d61b0f56e677
Valid from: Wed Dec 21 15:53:43 MSK 2022 until: Fri Dec 20 15:53:42 MSK 2024
Certificate fingerprints:
         SHA1: CA:76:66:E2:89:D8:01:4B:09:11:47:A9:D3:6F:21:EB:07:35:14:FD
         SHA256: 85:D6:7D:93:A9:BC:B4:1D:09:99:49:1E:28:AA:75:84:63:13:58:8B:A9:A4:43:BE:49:8A:7A:F4:D6:D9:74:11
...
BasicConstraints:[
  CA:false
  PathLen: undefined
]

I understand that this looks like mysticism, perhaps these are some features of how spring boot works in k8s. I only have one idea left: that certificates end up in the java trust store later than the application starts. But I don't know how to check this yet.

dmikusa commented 1 month ago

Yes, that all looks OK. I'm not really sure what else to add here.

I understand that this looks like mysticism, perhaps these are some features of how spring boot works in k8s.

Most of the time, when you're making HTTPS connections to things using the JVM, clients will use the default truststore for verification. That's where buildpacks load your certs, the default truststore. It is of course possible to customize most HTTPS clients and have them load trusted certificates from elsewhere, other truststore files. I believe the JVM TLS debug output in it's extreme verbosity will tell you where it's loaded the trusted certs from though, so if you suspect an issue you might try looking through those more to see what it says (I think it loads them once, when first needed so it may not be right by your actual request in the log output).

I only have one idea left: that certificates end up in the java trust store later than the application starts. But I don't know how to check this yet.

The way that buildpacks work, they contribute these exec.d helper binaries that go into the app image. The "launcher" is a binary that is executed when your app stops. The launcher has a few responsibilities, including setting env variables for you app and running the exec.d helpers. The exec.d helpers are what process your bindings and expose them to the system (OpenSSL) and the JVM.

In most situations, those will only run once. The launcher only runs once, when your container starts, so it only runs the exec.d helpers once, and it guarantees they all finish before your app starts running.

I say "in most situations", because if you trigger the "launcher" to run again, then it will run the exec.d helpers again. In that case, the helpers might run while your app is still running. There are two ways I've seen this happen.

  1. With the paketo-buildpacks/health-checker buildpack. Prior to version 2.0.0, we recommended folks use /cnb/process/health-check to run the health check, except this triggers the launcher to run the health check binary. Since the launcher runs, it also runs all of the exec.d helpers again and that caused the occasional TLS connection to fail if you happened to be making a connection during the small window when the certs were being reloaded.

  2. If you manually trigger the launcher, like if you docker exec or similar to enter the container and the command that you specify to run invokes the launcher. This is not usually what folks do. Folks usually run a shell and that won't trigger the launcher. I'm not sure why this might actually be done, idk maybe if you were trying to periodically invoke a second process in the container, like a cron job/schedule task, and the process you were invoking was one of the /cnb/process/* commands which would in turn invoke the launcher. That could do it.

Anyway, both of these are pretty rare, and I did look earlier in your build logs and didn't see the health-checker buildpack being used, so I don't think it's 1.). This probably isn't the issue, but just wanted to mention it and give some more context on how the certs get loaded.

paulc0te commented 1 month ago

Yes, these are not my options. I'll think about a workaround. Thank you for your time.