Open paulc0te opened 1 month ago
The certificates that I added during build stop working after I add the certificate at runtime.
Are you embedding the build-time certs into the image? I don't see that in your build output, but I wanted to double check.
To do that you would set BP_EMBED_CERTS=true
. By default, it doesn't to embed the ca certificates from build into the container so you'd need to also include them with your list of runtime ca certificates.
I'm still not 100% sure that will work, but that would be the first thing I'd try.
@dmikusa, thanks for your advice, the certificates actually made it into the container and are now here
ls /layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/
569e89cc.0 66990ff1.0 720381b8.0 8b90f171.0 8b90f171.1 8b90f171.2 921c95d1.0 a316e707.0 a316e707.1 a316e707.2 a316e707.3 a316e707.4
But this does not solve my problem, I need the java application to work after adding certificates. Now I'm getting the same error
[BackgroundTask-1-psz-admin] [MDC : jmixUser=psz-admin] c.a.minio.filestorage.MinioFileStorage :Error saving file to minio storage
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
This is part of the application start log
Added 1 additional CA certificate(s) to system truststore
Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58521, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
:: Spring Boot :: (v2.7.4)
2024-07-11 17:06:45.991 - INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication :Starting AtsArmApplication using Java 11.0.23 on arm-c8cbc5d76-kl2pb with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-07-11 17:06:45.996 - INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication :No active profile set, falling back to 1 default profile: "default"
2024-07-11 17:06:54.759 - INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Bootstrapping Spring Data Jmix repositories in DEFAULT mode.
2024-07-11 17:06:55.271 - INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Finished Spring Data repository scanning in 502 ms. Found 1 Jmix repository interfaces.
This is part of the build log
:bootBuildImage
> Pulling builder image 'site.com/paketobuildpacks/builder:0.3.158-base' ..................................................
> Pulled builder image 'site.com/paketobuildpacks/builder@sha256:baf8e13d21e274361a59557f717969f15794271e212145894bbff46a4173dc08'
> Pulling run image 'site.com/paketobuildpacks/run:1.2.49-base-cnb' ..................................................
> Pulled run image 'site.com/paketobuildpacks/run@sha256:3fbc7987519710abebafbfa67827b226a4b88630678c149a2dfb494cb73bd150'
> Pulling buildpack image 'site.com/paketobuildpacks/java:latest' ..................................................
> Pulled buildpack image 'site.com/paketobuildpacks/java@sha256:9f292e740e044c7c364bf5d7e666908f8a1860800d216212da7cd15e9e921bad'
> Executing lifecycle version v0.15.3
> Using build cache volume 'pack-cache-d871107f3bf9.build'
> Running creator
[creator] ===> ANALYZING
[creator] Previous image with name "site.com/ats/arm-base:D221" not found
[creator] ===> DETECTING
[creator] 6 of 26 buildpacks participating
[creator] paketo-buildpacks/ca-certificates 3.8.2
[creator] paketo-buildpacks/bellsoft-liberica 10.8.1
[creator] paketo-buildpacks/syft 1.47.1
[creator] paketo-buildpacks/executable-jar 6.10.1
[creator] paketo-buildpacks/dist-zip 5.8.1
[creator] paketo-buildpacks/spring-boot 5.30.1
[creator] ===> RESTORING
[creator] ===> BUILDING
..
[creator] Paketo Buildpack for CA Certificates 3.8.2
[creator] https://github.com/paketo-buildpacks/ca-certificates
[creator] Build Configuration:
[creator] $BP_EMBED_CERTS true Embed certificates into the image
[creator] $BP_ENABLE_RUNTIME_CERT_BINDING true Deprecated: Enable/disable certificate helper layer to add certs at runtime
[creator] $BP_RUNTIME_CERT_BINDING_DISABLED false Disable certificate helper layer to add certs at runtime
..
[creator] CA Certificates: Contributing to layer
[creator] Embedding CA certificate(s)
[creator] Added 12 additional CA certificate(s) to system truststore
Full build log here
Here I am retrieving all the added certs from the java truststore. It can be seen that all 13 certificates are in place
KEYTOOL=/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/keytool
CACERTS=/layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
$KEYTOOL -list \
-storepass $PASS \
-keystore $CACERTS
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
/tmp/ca-certificates1300581086/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
I also increased the JRE version to 11.0.23 and paketo-buildpacks/ca-certificates to 3.8.2
I would be grateful for any advice on what else I can try.
From everything you've show here, it all appears to be working. We can see the CA certs being added to the image, we can see the exec.d helper loading your additional ca cert at runtime. We can see the exec.d helper for the JVM running and pulling in 13 additional certificates to the JVM's truststore. That's everything it should be doing, and the certs should be accessible to your Java app at this point.
If it's still not trusting the connection, then you'd need to do some more investigation of the specific connection to see why it can't match up the certs, there could be a cert missing still. The advise here is helpful for doing that -> https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html
If you see something that looks off from what the buildpack is doing, let me know. Happy to look again, but I'm not seeing anything that looks incorrect with what you've posted here.
The certificates are indeed written to the java truststore, but the problem is that they stop working as soon as I add another one using the at runtime method. If I add all the same certificates, but during build, everything works. The certificates are all checked and working, the application is working.
In this regard, I have several questions: 1) Can these two methods of adding certificates work together (At build time, At runtime)? 2) I have a suspicion that helper, in At runtime mode, adds certificates either after the application starts loading or in parallel with it, which leads to the fact that some or all certificates become non-working. Since after adding to the java truststore, the application must be restarted. Can you refute or confirm this?
Thank you very much for the article, I will take a look at it.
Ok, thanks for expanding on that.
Can these two methods of adding certificates work together (At build time, At runtime)?
That sounds like the million-dollar question here. I'm not totally sure off-hand, but I'll take a look.
I have a suspicion that helper, in At runtime mode, adds certificates either after the application starts loading or in parallel with it, which leads to the fact that some or all certificates become non-working.
All of the exec.d helpers run before your app starts, so there's no need to restart the app. All of the certs are loaded before your app starts. It's weird because your cert does have all of the certs loaded both build + runtime when you list the output of the keystore file. That would leave me to believe it's doing the right thing.
Could you send me the same output for the keystore when you put all of the ca certs in at build time and your app works so I can compare them?
Thanks
I disabled SERVICE_BINDING_ROOT. I also had to make BP_EMBED_CERTS: false. Without this, my application still didn't work even when adding certificates during build. This is news to me.
Next, I built the application image using the mode of adding certificates during build. And it still works after this build. But I really want to configure all this in runtime mode.
Here is the application start log
Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58521, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
:: Spring Boot :: (v2.7.4)
2024-07-12 00:24:55.272 - INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication :Starting AtsArmApplication using Java 11.0.23 on arm-bf8b68589-56lnr with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-07-12 00:24:55.277 - INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication :No active profile set, falling back to 1 default profile: "default"
2024-07-12 00:25:04.777 - INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Bootstrapping Spring Data Jmix repositories in DEFAULT mode.
Here is part of the buildlog
> Task :bootBuildImage
:bootBuildImage
Building image 'site.com/ats/arm-base:D223'
> Pulling builder image 'site.com/paketobuildpacks/builder:0.3.158-base' ..................................................
> Pulled builder image 'site.com/paketobuildpacks/builder@sha256:baf8e13d21e274361a59557f717969f15794271e212145894bbff46a4173dc08'
> Pulling run image 'site.com/paketobuildpacks/run:1.2.49-base-cnb' ..................................................
> Pulled run image 'site.com/paketobuildpacks/run@sha256:3fbc7987519710abebafbfa67827b226a4b88630678c149a2dfb494cb73bd150'
> Pulling buildpack image 'site.com/paketobuildpacks/java:latest' ..................................................
> Pulled buildpack image 'site.com/paketobuildpacks/java@sha256:9f292e740e044c7c364bf5d7e666908f8a1860800d216212da7cd15e9e921bad'
> Executing lifecycle version v0.15.3
> Using build cache volume 'pack-cache-6e2b2907d17b.build'
> Running creator
[creator] ===> ANALYZING
[creator] Previous image with name "site.com/ats/arm-base:D223" not found
[creator] ===> DETECTING
[creator] 6 of 26 buildpacks participating
[creator] paketo-buildpacks/ca-certificates 3.8.2
[creator] paketo-buildpacks/bellsoft-liberica 10.8.1
[creator] paketo-buildpacks/syft 1.47.1
[creator] paketo-buildpacks/executable-jar 6.10.1
[creator] paketo-buildpacks/dist-zip 5.8.1
[creator] paketo-buildpacks/spring-boot 5.30.1
[creator] ===> RESTORING
[creator] ===> BUILDING
...
[creator] Paketo Buildpack for CA Certificates 3.8.2
[creator] https://github.com/paketo-buildpacks/ca-certificates
[creator] Build Configuration:
[creator] $BP_EMBED_CERTS false Embed certificates into the image
[creator] $BP_ENABLE_RUNTIME_CERT_BINDING true Deprecated: Enable/disable certificate helper layer to add certs at runtime
[creator] $BP_RUNTIME_CERT_BINDING_DISABLED false Disable certificate helper layer to add certs at runtime
...
[creator] CA Certificates: Contributing to layer
[creator] Added 13 additional CA certificate(s) to system truststore
...
[creator] Paketo Buildpack for BellSoft Liberica 10.8.1
[creator] https://github.com/paketo-buildpacks/bellsoft-liberica
[creator] Build Configuration:
[creator] $BP_JVM_JLINK_ARGS --no-man-pages --no-header-files --strip-debug --compress=1 configure custom link arguments (--output must be omitted)
[creator] $BP_JVM_JLINK_ENABLED false enables running jlink tool to generate custom JRE
[creator] $BP_JVM_TYPE JRE the JVM type - JDK or JRE
[creator] $BP_JVM_VERSION 11.* the Java version
[creator] Launch Configuration:
[creator] $BPL_DEBUG_ENABLED false enables Java remote debugging support
[creator] $BPL_DEBUG_PORT 8000 configure the remote debugging port
[creator] $BPL_DEBUG_SUSPEND false configure whether to suspend execution until a debugger has attached
[creator] $BPL_HEAP_DUMP_PATH write heap dumps on error to this path
[creator] $BPL_JAVA_NMT_ENABLED true enables Java Native Memory Tracking (NMT)
[creator] $BPL_JAVA_NMT_LEVEL summary configure level of NMT, summary or detail
[creator] $BPL_JFR_ARGS configure custom Java Flight Recording (JFR) arguments
[creator] $BPL_JFR_ENABLED false enables Java Flight Recording (JFR)
[creator] $BPL_JMX_ENABLED false enables Java Management Extensions (JMX)
[creator] $BPL_JMX_PORT 5000 configure the JMX port
[creator] $BPL_JVM_HEAD_ROOM 0 the headroom in memory calculation
[creator] $BPL_JVM_LOADED_CLASS_COUNT 35% of classes the number of loaded classes in memory calculation
[creator] $BPL_JVM_THREAD_COUNT 250 the number of threads in memory calculation
[creator] $JAVA_TOOL_OPTIONS the JVM launch flags
[creator] Using Java version 11.* from BP_JVM_VERSION
...
[creator] BellSoft Liberica JRE 11.0.23: Contributing to layer
[creator] Downloading from https://site.com/repository/elk-raw/bell-sw/Liberica/releases/download/11.0.23+12/bellsoft-jre11.0.23+12-linux-amd64.tar.gz
[creator] Verifying checksum
[creator] Writing Syft SBOM at /layers/paketo-buildpacks_bellsoft-liberica/jre.sbom.syft.json: {Artifacts:[{ID:a640c2f785dc95aa Name:BellSoft Liberica JRE Version:11.0.23 Type:UnknownPackage FoundBy:libpak Locations:[{Path:buildpack.toml}] Licenses:[GPL-2.0 WITH Classpath-exception-2.0] Language: CPEs:[cpe:2.3:a:oracle:jre:11.0.23:*:*:*:*:*:*:*] PURL:pkg:generic/bellsoft-jre@11.0.23?arch=amd64}] Source:{Type:directory Target:/layers/paketo-buildpacks_bellsoft-liberica/jre} Descriptor:{Name:syft Version:0.32.0} Schema:{Version:1.1.0 URL:https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json}}
[creator] Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
[creator] Adding 137 container CA certificates to JVM truststore
...
[creator] Paketo Buildpack for Spring Boot 5.30.1
[creator] https://github.com/paketo-buildpacks/spring-boot
[creator] Build Configuration:
[creator] $BPL_JVM_CDS_ENABLED false whether to enable CDS optimizations at runtime
[creator] $BPL_SPRING_AOT_ENABLED false whether to enable Spring AOT at runtime
[creator] $BP_JVM_CDS_ENABLED false whether to enable CDS & perform JVM training run
[creator] $BP_SPRING_AOT_ENABLED false whether to enable Spring AOT
[creator] $BP_SPRING_CLOUD_BINDINGS_DISABLED false whether to contribute Spring Boot cloud bindings support
[creator] $BP_SPRING_CLOUD_BINDINGS_VERSION 1 default version of Spring Cloud Bindings library to contribute
[creator] Launch Configuration:
[creator] $BPL_SPRING_CLOUD_BINDINGS_DISABLED false whether to auto-configure Spring Boot environment properties from bindings
[creator] $BPL_SPRING_CLOUD_BINDINGS_ENABLED true Deprecated - whether to auto-configure Spring Boot environment properties from bindings
[creator] Creating slices from layers index
[creator] dependencies (296.8 MB)
[creator] spring-boot-loader (258.2 KB)
[creator] snapshot-dependencies (0.0 B)
[creator] application (22.5 MB)
Here is the full buildlog in debug mode
This is the part of the trust store in which you can see all 13 added certificates during build. I replaced the ending with 11. I don’t know why))
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
For testing, I temporarily changed my infrastructure and now I don’t need ca-certificates when building the application. Now I am adding the same certificates but only at runtime. However, my application is still not working.
Just in case, I’ll clarify that my application is spring boot running in kubernetes. The directory with ca-certificates is copied inside the container via mount k8s secrets. There is also an env variable (SERVICE_BINDING_ROOT=/paketo-certs/platform/bindings) that points to the directory with ca-certificates. Among the certificates there are chains of certificates. Each part of the chain is divided into PEM files.
Here is the application start log
Added 13 additional CA certificate(s) to system truststore
Setting Active Processor Count to 12
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 4G, Thread Count: 250, Loaded Class Count: 58521, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,SPI,CLDR -Xms1024m -Xmx3200m -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=345138K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
:: Spring Boot :: (v2.7.4)
2024-07-12 14:55:04.956 - INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication :Starting AtsArmApplication using Java 11.0.23 on arm-c8cbc5d76-5gjw9 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-07-12 14:55:04.969 - INFO 1 --- [main] [MDC : ] com.ats.arm.AtsArmApplication :No active profile set, falling back to 1 default profile: "default"
2024-07-12 14:55:19.148 - INFO 1 --- [main] [MDC : ] .s.d.r.c.RepositoryConfigurationDelegate:Bootstrapping Spring Data Jmix repositories in DEFAULT mode.
Here is part of the buildlog
:bootBuildImage
Building image 'site.com/ats/arm-base:D224'
> Pulling builder image 'site.com/paketobuildpacks/builder:0.3.158-base' ..................................................
> Pulled builder image 'site.com/paketobuildpacks/builder@sha256:baf8e13d21e274361a59557f717969f15794271e212145894bbff46a4173dc08'
> Pulling run image 'site.com/paketobuildpacks/run:1.2.49-base-cnb' ..................................................
> Pulled run image 'site.com/paketobuildpacks/run@sha256:3fbc7987519710abebafbfa67827b226a4b88630678c149a2dfb494cb73bd150'
> Pulling buildpack image 'site.com/paketobuildpacks/java:latest' ..................................................
> Pulled buildpack image 'site.com/paketobuildpacks/java@sha256:9f292e740e044c7c364bf5d7e666908f8a1860800d216212da7cd15e9e921bad'
> Executing lifecycle version v0.15.3
> Using build cache volume 'pack-cache-961e92e2c9d2.build'
> Running creator
[creator] ===> ANALYZING
[creator] Previous image with name "site.com/ats/arm-base:D224" not found
[creator] ===> DETECTING
[creator] 6 of 26 buildpacks participating
[creator] paketo-buildpacks/ca-certificates 3.8.2
[creator] paketo-buildpacks/bellsoft-liberica 10.8.1
[creator] paketo-buildpacks/syft 1.47.1
[creator] paketo-buildpacks/executable-jar 6.10.1
[creator] paketo-buildpacks/dist-zip 5.8.1
[creator] paketo-buildpacks/spring-boot 5.30.1
[creator] ===> RESTORING
[creator] ===> BUILDING
[creator] Paketo Buildpack for CA Certificates 3.8.2
[creator] https://github.com/paketo-buildpacks/ca-certificates
[creator] Build Configuration:
[creator] $BP_EMBED_CERTS false Embed certificates into the image
[creator] $BP_ENABLE_RUNTIME_CERT_BINDING true Deprecated: Enable/disable certificate helper layer to add certs at runtime
[creator] $BP_RUNTIME_CERT_BINDING_DISABLED false Disable certificate helper layer to add certs at runtime
[creator] Paketo Buildpack for BellSoft Liberica 10.8.1
[creator] https://github.com/paketo-buildpacks/bellsoft-liberica
[creator] Build Configuration:
[creator] $BP_JVM_JLINK_ARGS --no-man-pages --no-header-files --strip-debug --compress=1 configure custom link arguments (--output must be omitted)
[creator] $BP_JVM_JLINK_ENABLED false enables running jlink tool to generate custom JRE
[creator] $BP_JVM_TYPE JRE the JVM type - JDK or JRE
[creator] $BP_JVM_VERSION 11.* the Java version
[creator] Launch Configuration:
[creator] $BPL_DEBUG_ENABLED false enables Java remote debugging support
[creator] $BPL_DEBUG_PORT 8000 configure the remote debugging port
[creator] $BPL_DEBUG_SUSPEND false configure whether to suspend execution until a debugger has attached
[creator] $BPL_HEAP_DUMP_PATH write heap dumps on error to this path
[creator] $BPL_JAVA_NMT_ENABLED true enables Java Native Memory Tracking (NMT)
[creator] $BPL_JAVA_NMT_LEVEL summary configure level of NMT, summary or detail
[creator] $BPL_JFR_ARGS configure custom Java Flight Recording (JFR) arguments
[creator] $BPL_JFR_ENABLED false enables Java Flight Recording (JFR)
[creator] $BPL_JMX_ENABLED false enables Java Management Extensions (JMX)
[creator] $BPL_JMX_PORT 5000 configure the JMX port
[creator] $BPL_JVM_HEAD_ROOM 0 the headroom in memory calculation
[creator] $BPL_JVM_LOADED_CLASS_COUNT 35% of classes the number of loaded classes in memory calculation
[creator] $BPL_JVM_THREAD_COUNT 250 the number of threads in memory calculation
[creator] $JAVA_TOOL_OPTIONS the JVM launch flags
[creator] Using Java version 11.* from BP_JVM_VERSION
[creator] BellSoft Liberica JRE 11.0.23: Contributing to layer
[creator] Downloading from http://site.com:8081/repository/elk-raw/bell-sw/Liberica/releases/download/11.0.23+12/bellsoft-jre11.0.23+12-linux-amd64.tar.gz
[creator] Verifying checksum
[creator] Writing Syft SBOM at /layers/paketo-buildpacks_bellsoft-liberica/jre.sbom.syft.json: {Artifacts:[{ID:a640c2f785dc95aa Name:BellSoft Liberica JRE Version:11.0.23 Type:UnknownPackage FoundBy:libpak Locations:[{Path:buildpack.toml}] Licenses:[GPL-2.0 WITH Classpath-exception-2.0] Language: CPEs:[cpe:2.3:a:oracle:jre:11.0.23:*:*:*:*:*:*:*] PURL:pkg:generic/bellsoft-jre@11.0.23?arch=amd64}] Source:{Type:directory Target:/layers/paketo-buildpacks_bellsoft-liberica/jre} Descriptor:{Name:syft Version:0.32.0} Schema:{Version:1.1.0 URL:https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-1.1.0.json}}
[creator] Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
[creator] Adding 124 container CA certificates to JVM truststore
[creator] Paketo Buildpack for Spring Boot 5.30.1
[creator] https://github.com/paketo-buildpacks/spring-boot
[creator] Build Configuration:
[creator] $BPL_JVM_CDS_ENABLED false whether to enable CDS optimizations at runtime
[creator] $BPL_SPRING_AOT_ENABLED false whether to enable Spring AOT at runtime
[creator] $BP_JVM_CDS_ENABLED false whether to enable CDS & perform JVM training run
[creator] $BP_SPRING_AOT_ENABLED false whether to enable Spring AOT
[creator] $BP_SPRING_CLOUD_BINDINGS_DISABLED false whether to contribute Spring Boot cloud bindings support
[creator] $BP_SPRING_CLOUD_BINDINGS_VERSION 1 default version of Spring Cloud Bindings library to contribute
[creator] Launch Configuration:
[creator] $BPL_SPRING_CLOUD_BINDINGS_DISABLED false whether to auto-configure Spring Boot environment properties from bindings
[creator] $BPL_SPRING_CLOUD_BINDINGS_ENABLED true Deprecated - whether to auto-configure Spring Boot environment properties from bindings
Here is the full buildlog in debug mode
This is the part of the trust store in which you can see all 13 added certificates at runtime
...
/tmp/ca-certificates2357141951/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
/tmp/ca-certificates2357141951/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
/tmp/ca-certificates2357141951/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
/tmp/ca-certificates2357141951/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
/tmp/ca-certificates2357141951/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
/tmp/ca-certificates2357141951/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
/tmp/ca-certificates2357141951/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
/tmp/ca-certificates2357141951/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
/tmp/ca-certificates2357141951/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
/tmp/ca-certificates2357141951/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
/tmp/ca-certificates2357141951/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
/tmp/ca-certificates2357141951/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
/tmp/ca-certificates2357141951/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
...
Any ideas why this is happening?
Added 13 additional CA certificate(s) to system truststore Adding 137 container CA certificates to JVM truststore
Well, I don't see any change to the number of certificates being loaded. It seems the same as when you had some being loaded at build and some at runtime. That's what I would expect to see.
I also did a comparison of the keystore output from your build and from your runtime and they look the same to me. The only difference is in the path from which it imported the original certs. That is to be expected and that's not going to matter. The certificate fingerprints are all the same and in the same order, that's the crucial part.
The trouble is that neither of those examples actually resulted in a working app. What I'd suggest doing is comparing the state of the keystore when it's not working with one that is working. It's likely that something is missing or different and if we can figure out what that is then we can figure out why it's missing and how to fix that.
So if you keytool list
the truststore that's generated in a container where your app works and then do the same thing in the container where it doesn't work, take the outputs and diff -u
them (or use your diff util of choice). It's very likely that something will be different.
Below is a comparison 2 files don't work and working. The only difference is in aliases. There are the same number of certificates, 295, they are added in the same order and they have the same fingerprint.
Certificate information was retrieved using the command
KEYTOOL=/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/keytool
CACERTS=/layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
$KEYTOOL -list \
-storepass password \
-keystore $CACERTS
diff -u cacerts_didnot_work.txt cacerts_worked.txt
--- cacerts_didnot_work.txt 2024-07-12 09:32:18.000000000 -0400
+++ cacerts_worked.txt 2024-07-12 10:35:37.000000000 -0400
@@ -253,31 +253,31 @@
Certificate fingerprint (SHA-256): 2C:E1:CB:0B:F9:D2:F9:E1:02:99:3F:BE:21:51:52:C3:B2:DD:0C:AB:DE:1C:68:E5:31:9B:83:91:54:DB:B7:11
/etc/ssl/certs/ca-certificates.crt-99, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 56:8D:69:05:A2:C8:87:08:A4:B3:02:51:90:ED:CF:ED:B1:97:4A:60:6A:13:C6:E5:29:0F:CB:2A:E6:3E:DA:11
-/tmp/ca-certificates2357141951/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/569e89cc.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 03:5B:7E:AD:50:4C:7C:17:29:50:35:54:DA:A7:14:15:87:3B:BA:49:0A:3C:60:56:42:C7:12:B6:AF:9F:DC:11
-/tmp/ca-certificates2357141951/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/66990ff1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
-/tmp/ca-certificates2357141951/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/720381b8.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
-/tmp/ca-certificates2357141951/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): A6:14:07:AE:1D:23:6C:05:CE:9A:91:96:EE:64:5C:8E:A7:2A:BF:61:A2:08:3E:14:8D:54:E4:75:E0:31:D4:11
-/tmp/ca-certificates2357141951/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 0A:58:20:0D:21:52:AA:BF:1D:CD:7D:D4:BE:04:64:5A:91:18:CA:80:03:E1:CA:13:55:21:F4:7D:8E:68:96:11
-/tmp/ca-certificates2357141951/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): DF:FD:0E:06:F3:66:8B:15:B2:2A:23:C2:20:C7:1C:4B:C2:76:D8:81:78:01:B0:F2:83:FC:D9:AA:75:11:B7:11
-/tmp/ca-certificates2357141951/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/8b90f171.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:DD:B9:80:B9:65:53:9C:F6:A4:E9:ED:1E:61:A0:DE:8A:26:B3:B5:84:05:6F:62:49:6B:D3:AA:4B:17:6A:11
-/tmp/ca-certificates2357141951/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/921c95d1.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 92:18:46:62:6E:63:EB:B6:D3:26:86:56:21:DA:AD:6E:CA:4A:E0:77:CA:86:39:DE:08:B7:11:1C:95:E2:BA:11
-/tmp/ca-certificates2357141951/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.0-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): F5:E3:3B:57:B6:16:DF:A3:6D:2B:5C:B3:74:57:68:B0:71:DE:75:D0:8A:4C:5C:7D:9C:84:05:71:2B:AD:85:11
-/tmp/ca-certificates2357141951/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.1-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 75:47:C7:34:3A:28:B4:17:42:F3:B9:3A:FA:B3:03:AD:E4:73:32:26:9E:35:86:23:74:12:8F:DB:DF:0C:78:11
-/tmp/ca-certificates2357141951/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.2-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 8D:23:4D:B6:43:09:CD:1D:19:29:E9:77:9A:E4:63:18:43:F1:E3:A3:17:57:01:66:C9:1A:90:04:0B:18:2B:11
-/tmp/ca-certificates2357141951/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.3-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): 6E:34:72:4F:70:9C:29:79:8F:A7:0D:27:0F:5D:D1:13:B2:F6:32:B3:64:9A:37:AC:2B:C0:AF:95:9A:3E:B5:11
-/tmp/ca-certificates2357141951/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
+/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/a316e707.4-0, Jan 1, 1980, trustedCertEntry,
Certificate fingerprint (SHA-256): BA:FF:F6:56:47:30:A4:C7:2C:F0:1D:C9:57:97:5C:26:27:F6:7B:37:34:4C:1A:00:1F:AC:E9:6B:EB:75:23:11
actalisauthenticationrootca [jdk], Sep 22, 2011, trustedCertEntry,
Certificate fingerprint (SHA-256): 55:92:60:84:EC:96:3A:64:B9:6E:2A:BE:01:CE:0B:A8:6A:64:FB:FE:BC:C7:AA:B5:AF:C1:55:B3:7F:D7:60:11
If they're all the same, that sounds to me like the buildpack is doing the right thing, or at least it's behaving consistently in both scenarios. Ultimately, its job is to take the ca-certs from your bindings, add them to the system store (i.e. make OpenSSL see them), and then add them to the JVM's default keystore, which makes the JVM see them. In both cases, we can see them in the JVM's truststore, the ultimate destination, so it appears to be doing its job.
Beyond that, we'd have to know specifically why it's failing to trust the certificate. If you enable the TLS debugging features in the JVM, you can usually get a pretty good idea of why it's failing.
https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html
If it's failing because of something the buildpack is doing, we can certainly address that, but without knowing why the JVM is rejecting it, it's hard for me to comment further.
enabled debug for the application. This is all that is visible in him at the moment of the error
javax.net.ssl|DEBUG|45 E3|BackgroundTask-0-psz-admin|2024-07-12 20:23:17.707 MSK|null:-1|Consuming server Certificate handshake message (
"Certificate": {
"certificate_request_context": "",
"certificate_list": [
{
"certificate" : {
"version" : "v3",
"serial number" : "5D 60 2C CB D4 E8 07 B1 06 9B 82 A4 77 D1 D6 1B 0F 56 E6 11",
"signature algorithm": "SHA256withRSA",
"issuer" : "CN=ElkSubCA",
"not before" : "2022-12-21 15:53:43.000 MSK",
"not after" : "2024-12-20 15:53:42.000 MSK",
"subject" : "CN=site.com",
"subject public key" : "RSA",
"extensions" : [
{
ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 25 74 EE 7A 70 70 CB E3 6E 36 C1 AC FE E8 D5 15 %t.zpp..n6......
0010: 43 A9 BA C6 C...
]
]
},
{
ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:false
PathLen: undefined
]
},
{
ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
]
},
{
ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]
},
{
ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: site.com
DNSName: site.com00.test.elk.loc
DNSName: site.com01.test.elk.loc
DNSName: site.com02.test.elk.loc
DNSName: site.com03.test.elk.loc
IPAddress: 192.168.1.33
IPAddress: 192.168.1.34
IPAddress: 192.168.1.35
IPAddress: 192.168.1.36
IPAddress: 192.168.1.37
]
},
{
ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 9F 57 13 8C B6 49 3E 65 65 3E A0 FD FB 45 65 DC .W...I>ee>...Ee.
0010: 9D 60 06 43 .`.C
]
]
}
]}
"extensions": {
<no extension>
}
},
]
}
)
javax.net.ssl|DEBUG|45 E3|BackgroundTask-0-psz-admin|2024-07-12 20:23:17.707 MSK|null:-1|Ignore unavailable extension: status_request
javax.net.ssl|ERROR|45 E3|BackgroundTask-0-psz-admin|2024-07-12 20:23:17.710 MSK|null:-1|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (
"throwable" : {
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
The PKIX path error is saying that it can't find a path from the server's cert to a trusted cert in the JVMs truststore. You can walk that path on your own if you start with the issuer of the server's cert, which is "issuer": "CN=ElkSubCA",
. Then look for a cert in the truststore where the subject is CN=ElkSubCA
. In a nutshell, it's saying it can't find one.
If that doesn't exist in the truststore, check your bindings and confirm you have it in your bindings. If it's in your bindings, but not in the truststore, then we can look at the bindings to see why it might be skipped.
All necessary certificates are available in the java trust store. The application works when I add them to the image at the build stage. The application does not work when I add the same certificates in at runtime mode.
The certificate chain for a host with which SSL does not work looks like this
ATSRootCA
ElkSubCA
site.com <-- host
Look, all these certificates are in the java truststore
The command with which I retrieve information about certificates
KEYTOOL=/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/keytool
CACERTS=/layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
$KEYTOOL -list \
-v \
-storepass $PASS \
-keystore $CACERTS \
-alias $CERT_ALIASE
Alias name: /tmp/ca-certificates8216640/66990ff1.0-0
Creation date: Jan 1, 1980
Entry type: trustedCertEntry
Owner: CN=ATSRootCA
Issuer: CN=ATSRootCA
Serial number: 572e997043ba54b649c85b8aefdc3211
Valid from: Wed Jan 21 15:27:47 MSK 2009 until: Sun Jan 21 15:38:47 MSK 2029
Certificate fingerprints:
SHA1: 7D:7F:07:9F:E5:70:0D:FB:C0:F0:49:50:F7:EC:84:E0:D3:D4:55:06
SHA256: 3C:0E:D2:2F:5C:AA:30:59:58:1C:B1:21:6E:4C:17:4E:50:98:66:92:75:E4:3F:65:7B:BB:B4:AA:E5:07:A0:11
...
BasicConstraints:[
CA:true
PathLen:2147483647
]
Alias name: /tmp/ca-certificates8216640/720381b8.0-0
Creation date: Jan 1, 1980
Entry type: trustedCertEntry
Owner: CN=ElkSubCA
Issuer: CN=ATSRootCA
Serial number: 47000000366403e8395a9925a3000200000036
Valid from: Wed Dec 14 22:26:39 MSK 2022 until: Sun Jan 21 15:38:47 MSK 2029
Certificate fingerprints:
SHA1: 32:F3:84:3A:DA:77:81:96:0A:A3:47:4F:8C:C3:35:52:04:17:67:57
SHA256: 5A:D5:EB:59:27:CF:78:92:B2:F8:2F:73:BA:C8:19:97:CE:C0:F9:96:A7:20:D5:ED:8C:5B:56:AE:E9:3A:9A:11
...
BasicConstraints:[
CA:true
PathLen:2147483647
]
Alias name: /tmp/ca-certificates8216640/436e719d.0-0
Creation date: Jan 1, 1980
Entry type: trustedCertEntry
Owner: CN=site.com
Issuer: CN=ElkSubCA
Serial number: 5d602ccbd4e807b1069b82a477d1d61b0f56e677
Valid from: Wed Dec 21 15:53:43 MSK 2022 until: Fri Dec 20 15:53:42 MSK 2024
Certificate fingerprints:
SHA1: CA:76:66:E2:89:D8:01:4B:09:11:47:A9:D3:6F:21:EB:07:35:14:FD
SHA256: 85:D6:7D:93:A9:BC:B4:1D:09:99:49:1E:28:AA:75:84:63:13:58:8B:A9:A4:43:BE:49:8A:7A:F4:D6:D9:74:11
...
BasicConstraints:[
CA:false
PathLen: undefined
]
I understand that this looks like mysticism, perhaps these are some features of how spring boot works in k8s. I only have one idea left: that certificates end up in the java trust store later than the application starts. But I don't know how to check this yet.
Yes, that all looks OK. I'm not really sure what else to add here.
I understand that this looks like mysticism, perhaps these are some features of how spring boot works in k8s.
Most of the time, when you're making HTTPS connections to things using the JVM, clients will use the default truststore for verification. That's where buildpacks load your certs, the default truststore. It is of course possible to customize most HTTPS clients and have them load trusted certificates from elsewhere, other truststore files. I believe the JVM TLS debug output in it's extreme verbosity will tell you where it's loaded the trusted certs from though, so if you suspect an issue you might try looking through those more to see what it says (I think it loads them once, when first needed so it may not be right by your actual request in the log output).
I only have one idea left: that certificates end up in the java trust store later than the application starts. But I don't know how to check this yet.
The way that buildpacks work, they contribute these exec.d helper binaries that go into the app image. The "launcher" is a binary that is executed when your app stops. The launcher has a few responsibilities, including setting env variables for you app and running the exec.d helpers. The exec.d helpers are what process your bindings and expose them to the system (OpenSSL) and the JVM.
In most situations, those will only run once. The launcher only runs once, when your container starts, so it only runs the exec.d helpers once, and it guarantees they all finish before your app starts running.
I say "in most situations", because if you trigger the "launcher" to run again, then it will run the exec.d helpers again. In that case, the helpers might run while your app is still running. There are two ways I've seen this happen.
With the paketo-buildpacks/health-checker buildpack. Prior to version 2.0.0, we recommended folks use /cnb/process/health-check
to run the health check, except this triggers the launcher to run the health check binary. Since the launcher runs, it also runs all of the exec.d helpers again and that caused the occasional TLS connection to fail if you happened to be making a connection during the small window when the certs were being reloaded.
If you manually trigger the launcher, like if you docker exec
or similar to enter the container and the command that you specify to run invokes the launcher. This is not usually what folks do. Folks usually run a shell and that won't trigger the launcher. I'm not sure why this might actually be done, idk maybe if you were trying to periodically invoke a second process in the container, like a cron job/schedule task, and the process you were invoking was one of the /cnb/process/*
commands which would in turn invoke the launcher. That could do it.
Anyway, both of these are pretty rare, and I did look earlier in your build logs and didn't see the health-checker buildpack being used, so I don't think it's 1.). This probably isn't the issue, but just wanted to mention it and give some more context on how the certs get loaded.
Yes, these are not my options. I'll think about a workaround. Thank you for your time.
Expected Behavior
Adding certificates at runtime must not break work of certificates added during build.
Current Behavior
The certificates that I added during build stop working after I add the certificate at runtime.
Possible Solution
If I add the same certificates only during build, then everything works
Steps to Reproduce
I'm building a spring boot application image using a gradle task that runs paketo-buildpacks. After build, the application works in k8s.
Directory with certificates when building an image
gradle task with which the application image is built
The command I run the build with
Part of the Buildlog that I see when building
Attached is the full build_log
The env variable which points to the directory in which the ca-certificates directory is located
Directory with certificates in a container
Application start log
After the application starts, I try to use certificates, but they don't work. I am getting the following error
_JAVAHOME points to the correct cacerts location
All certificates are recorded in cacerts and are working. I checked this by exporting and using curl.
I noticed that the certificates added in cacerts during build look like this
Certificates added at runtime look like this
I tried updating paketo-buildpacks/ca-certificates to the latest version but it did not solve the problem
Motivations
I need to use 2 of these methods - because during the build phase paketo-buildpacks accesses the https repository for its dependencies. Accordingly, he should already have a certificate. What is the reason for this behavior of paketo-buildpacks/ca-certificates ?