Closed TomaszKlosinski closed 4 years ago
Hi @TomaszKlosinski,
What is the result of the following?
$ cat <<EOF > A.java
public class A {
public static void main(String[] args) throws Exception {
new java.net.URL("https://gcr.io/v2/").openStream().close();
}
}
EOF
$ java -Djavax.net.ssl.trustStoreType=KeychainStore A.java
If it works, then can you run the code with Maven?
Hello @chanseokoh , here's the result:
Exception in thread "main" javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at java.base/java.net.URL.openStream(URL.java:1140)
at A.main(A.java:3)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:629)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at java.base/java.net.URL.openStream(URL.java:1140)
at A.main(A.java:3)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:404)
at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:179)
at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:119)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 30 more
Hmmm... I guess your OSX keychain is broken or empty? gcr.io
is a Google server, and no one should have trouble verifying the server. Apparently, this is not a Jib issue, but a general issue that your JVM+Keychain cannot verify trivially obvious servers. Maybe ask someone who are familiar with Java on Mac?
Maybe the problem is related to my Java version. I've downloaded couple of them using brew. So the one I'm using now is:
echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
It's sure not broken or empty, because I can use https in Chrome (which uses Keychain) and in docker (I can push the image to my repo). It doesn't work only with Java.
Maybe ask someone who are familiar with Java on Mac?
Ok, I'll try on Stackoverflow. Thanks for the help.
No problem. I'd appreciate your update once you figure out the cause.
I did one more interesting test. I've imported the cert into Java cacerts:
sudo ${JAVA_HOME}/bin/keytool -import -trustcacerts -keystore ${JAVA_HOME}/lib/security/cacerts -storepass changeit -noprompt -alias mycompany -file mycompany.crt
Then I did your test for gcr.io and for my company's registry:
$ java -Djavax.net.ssl.trustStore=${JAVA_HOME}/lib/security/cacerts B.java
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 400 for URL: https://docker-repo.bananas.xyz/
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1919)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at java.base/java.net.URL.openStream(URL.java:1140)
at A.main(B.java:3)
$ java -Djavax.net.ssl.trustStore=${JAVA_HOME}/lib/security/cacerts A.java
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 401 for URL: https://gcr.io/v2/
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1919)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at java.base/java.net.URL.openStream(URL.java:1140)
at A.main(A.java:3)
So they seem to respond correctly. But when I try to build and push the image, then I get yet another error:
$ mvn -B compile com.google.cloud.tools:jib-maven-plugin:build -Djavax.net.ssl.trustStore=${JAVA_HOME}/lib/security/cacerts
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.example:spring-boot >-----------------------
[INFO] Building spring-boot 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ spring-boot ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/tomasz/Development/skaffold-java-example/src/main/resources
[INFO] skip non existing resourceDirectory /Users/tomasz/Development/skaffold-java-example/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ spring-boot ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- jib-maven-plugin:2.2.0:build (default-cli) @ spring-boot ---
[INFO]
[INFO] Containerizing application to docker-repo.bananas.xyz/sysadmin/skaffold-java-example...
[WARNING] Base image 'gcr.io/distroless/java:11-debug' does not use a specific image digest - build may not be reproducible
[INFO] Getting manifest for base image gcr.io/distroless/java:11-debug...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
[INFO] Using credentials from Docker config (/Users/tomasz/.docker/config.json) for docker-repo.bananas.xyz/sysadmin/skaffold-java-example
[ERROR] I/O error for image [gcr.io/distroless/java]:
[ERROR] javax.net.ssl.SSLException
[ERROR] Unexpected rethrowing
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.132 s
[INFO] Finished at: 2020-05-21T00:22:11+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.2.0:build (default-cli) on project spring-boot: Build image failed: Failed to authenticate with registry docker-repo.bananas.xyz/sysadmin/skaffold-java-example because: insecure HTTP connection not allowed: http://docker-repo.bananas.xyz/v2/token?service=http://docker-repo.bananas.xyz/v2/token&scope=repository:sysadmin/skaffold-java-example:pull,push -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
This make me really confused. I'll try to repeat these tests on a Linux box and I'll let you know if I had any success. Cheers!
Interesting. BTW, if you've modified and replaced the default lib/security/cacerts
, I believe you don't need to set -Djavax.net.ssl.trustStore=${JAVA_HOME}/lib/security/cacerts
. Maybe try without it. Importing a server certificate should normally work. Also pass -X
to Maven to get the full stack trace.
Yes, it's really weird. The second option definitely should work.
Here's the log with full stack trace:
Ah, it's working actually. The Maven build was able to verify docker-repo.bananas.xyz
. It's just that your registry docker-repo.bananas.xyz
is (mis-)configured to direct Jib to go to http://docker-repo.bananas.xyz/v2/token
during bearer token authentication. (Mind http://
.) For example, if you follow these instructions to get raw HTTP responses from your registry, you'll be able to verify that your registry returns a header like
WWW-Authenticate: Bearer realm=http://... service=http://...
You need to reconfigure your registry to return HTTPS. Or, just get rid of the http://
prefix.
I'm still curious why -Djavax.net.ssl.trustStoreType=KeychainStore
doesn't work. I'd appreciate your update once you figure it out.
Thanks for the help! We're using Sontatype Nexus as Docker registry and we used pretty much default configuration. I'll look into their docs to figure out how to reconfigure it to actually redirect to https. I'll let you. know if I had any progress with the KeychainStore. Cheers!
Update 1: I've just found out that the Nexus is using HTTP and we have a reverse proxy in front of it to do the SSL offloading. This is probably what is causing the issue.
Update 2: We've reconfigured Neuxs to use HTTPS and it works now. Thanks for the help!
@TomaszKlosinski have you figured out why -Djavax.net.ssl.trustStoreType=KeychainStore
doesn't work?
Hi @chanseokoh , no, unfortunately, I have no clue. I thought it's maybe OpenJDK vs Oracle Java thing, but it didn't work on both versions for me. Somewhere I found information that with KeychainStore you can't access all your certificates and passwords (only those that are assigned to your user), it's somehow limited but I'm not sure how it works exactly.
PS. I've just opened a question on StackOverflow, maybe somebody will provide some clues.
Environment:
Description of the issue: I'm trying to build an image (based on
gcr.io/distroless/java:11-debug
) and push it to my company's private registry.It works if I ignore TLS:
It works over https if I build and push identical image with a Dockerfile.
But when I try to force the maven/jib to pick up the certificates from the Keychain (using
-Djavax.net.ssl.trustStoreType=KeychainStore
), then it fails, complaining that it can't get the certificate for gcr.io. I don't use any proxy, so I don't understand why it doesn't work - just like with Dockerfile.Expected behavior: Maven/Jib should pick up the certificate and the image should be pushed to the registry over https.
Steps to reproduce:
jib-maven-plugin
Configuration:Log output:
Additional information Dockerfile that works fine:
Thanks in advance for help.