Closed DidierSchonne closed 6 years ago
Hi @DidierSchonne,
For the issue about 403 Forbidden, the log seems to show that the username and the password in you ~/.m2/settings.xml
are not being picked up. What's the contents of your settings.xml
? I expect it would look like
<settings>
...
<servers>
...
<server>
<id>gitlab.mydomain.com:4567</id>
<username>...</username>
<password>...</password>
</server>
</servers>
</settings>
Hi @chanseokoh, I tried with and without the port and I still have the same issue, and it is well filled as you expect :
<servers>
<server>
<id>gitlab.mydomain.com:4567</id>
<username>***</username>
<password>***</password>
</server>
</servers>
I tried many times to copy / paste from a docker login command in order to avoid a misspelling. And don't understand why I have this: No credentials could be retrieved for registry gitlab.mydomain.com:4567
Hi @DidierSchonne , if you run mvn help:effective-settings
, do the settings for your registry show up in the output?
I tried with and without the port
FYI, the port part (:4567
) is necessary.
And don't understand why I have this:
No credentials could be retrieved for registry gitlab.mydomain.com:4567
This is the sign that Jib was not able to find the credentials from your settings.xml
. Let's check the output of mvn help:effective-settings
as @coollog suggested. FYI, you can do mvn --settings /path/to/custom/settings-xml.file ...
to provide any arbitrary settings file.
The other thing to try is to enable full-auth logging. Follow the instructions for enabling logging with the Google HTTP Client, but instead of "CONFIG" use "ALL". That should show the username and password being passed up to gitlab.
@coollog, @chanseokoh, result is:
<settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/home/did/.m2/repository</localRepository>
<servers xmlns="http://maven.apache.org/SETTINGS/1.1.0">
<server>
<username>...</username>
<password>***</password>
<id>gitlab.mydomain.com:4567</id>
</server>
</servers>
<pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0">
<pluginGroup>org.apache.maven.plugins</pluginGroup>
<pluginGroup>org.codehaus.mojo</pluginGroup>
</pluginGroups>
</settings>
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
Thank you @briandealwis, I'm going to try your suggestion.
@DidierSchonne and can you confirm you still see No credentials could be retrieved for registry gitlab.mydomain.com:4567
?
Do either of your username or password contain any XML-relevant characters? Enabling Google HTTP logging should help reveal that. Basically you should see an item like:
Jul 31, 2018 12:24:56 PM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST --------------
GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:briandealwis/first:pull,push
Accept: */*
Accept-Encoding: gzip
Authorization: Basic YnJpYW5kZWFsd2lzOmZvb2Jhcg==
User-Agent: Google-HTTP-Java-Client/1.23.0 (gzip)
but to your gitlab instance. Base64-decoding that Authorization blog should reveal your username and password, separated with a colon.
$ echo 'YnJpYW5kZWFsd2lzOmZvb2Jhcg==' | base64 -D
briandealwis:foobar
@chanseokoh, I confirm and still have it:
[INFO] Retrieving registry credentials for gitlab.mydomain.com:4567...
[DEBUG] TIMING Retrieving registry credentials for gitlab.mydomain.com:4567
[DEBUG] RUNNING Retrieving registry credentials for gitlab.mydomain.com:4567
[DEBUG] No credentials could be retrieved for registry gitlab.mydomain.com:4567
[DEBUG] TIMED Retrieving registry credentials for gitlab.mydomain.com:4567 : 1.573 ms
[DEBUG] Retrieving registry credentials for gitlab.mydomain.com:4567 : 1.573 ms
[DEBUG] TIMING Authenticating with push to gitlab.mydomain.com:4567
[DEBUG] RUNNING Authenticating with push to gitlab.mydomain.com:4567
@briandealwis, my password contained a special character but I changed my password to avoid any issue with it. I'm currently checking what the logs have to say.
@briandealwis Using your tip, the authorization header is simply not present.
Oh, I suspect you're communicating over HTTP, and not HTTPS. We don't send auth over HTTP by default, since it's insecure. You'll need to run with the sendCredentialsOverHttp
system property set to true (#599). This isn't a recommended approach: you should configure your registry with an SSL certificate, even if self-signed.
@briandealwis I'm not sure if that's the case; if that were true, jib should log something like "Required credentials for xxxx/yyyy were not sent because the connection was over HTTP", but I don't see that anywhere in their logs. If that is the case here, though, that may be a bug with jib not displaying the correct error message.
Hmm.. I added SSL through Let's Encrypt to Gitlab and parameterized the Gitlab Registry to use it. If I try to connect to http://gitlab.mydomain.com:4567 I'm switched to https.
@DidierSchonne I noticed that you're running mvn
with sudo
. Regarding https://github.com/GoogleContainerTools/jib/issues/746#issuecomment-409276911, did you do
mvn help:effective-setting
or
sudo mvn help:effective-setting
? Do they give different results? For sudo mvn ...
, I believe it will look into, e.g., /root/.m2/settings.xml
on a typical Linux. Most likely you shouldn't do sudo
when building.
@DidierSchonne I see from gitlab's documentation that it supports the use of docker login
. Perhaps try that instead of the Maven settings: we should pick up the settings from the ~/.docker/config.json
.
Users should now be able to login to the Container Registry with their GitLab credentials using:
docker login gitlab.example.com:4567
You're right @chanseokoh, with sudo I don't have my right settings. So I added my user to the Docker group, because I guess that it is needed to push to the registry, right?
But with mvn compile -X com.google.cloud.tools:jib-maven-plugin:0.9.7:build
I have this error: Caused by: com.google.cloud.tools.jib.registry.InsecureRegistryException: Only secure connections are allowed, but tried to reach URL http://gcr.io/v2/distroless/java/manifests/latest
which I didn't have with sudo.
And I'm stuck before trying to push to the registry.
So I added my user to the Docker group, because I guess that it is needed to push to the registry, right?
Jib doesn't need privileged rights to push an image to a remote registry (unless you are pushing to your local Docker daemon, which usually requires sudo
). BTW, I don't think adding yourself to the Docker group makes mvn
without sudo
to look into /root/.m2/settings.xml
. It'll look into your ~/.m2/settings.xml
, so it's enough to have it correct. In any case, Jib should work without sudo
, so I'd forget about the Docker group; it's irrelevant to the InsecureRegistryException
issue you mentioned. Jib even works without docker
locally installed (when pushing to a remote registry).
That said, I'll look into the log and try to think what's happening with the following error.
Only secure connections are allowed, but tried to reach URL: http://gcr.io/v2/distroless/java/manifests/lates
I am having same issue similar to DidierSchonne
Failed to execute goal com.google.cloud.tools:jib-maven-plugin:0.9.7:build (default-cli) on project SkuGroupMaintainWS: Build image failed, perhaps you should use a registry that supports HTTPS or set the configuration parameter 'allowInsecureRegistries': Only secure connections are allowed, but tried to reach URL http://us.gcr.io/v2/xxx/xxxx
@adorearun yeah, it looks very similar. Interesting. Can you tell us which base and target registries you are using?
@DidierSchonne and @adorearun are you behind proxies by any chance?
GCR is our target registry and I am using maven plugin. It was working earlier only failing now.
@briandealwis Nope. But It was strange for me because it used to work after I upgrade from 0.9.4 version to 0.9.7 in the morning and after an hour it started failing. Does it ring any bell?
@chanseokoh @briandealwis To debug further if I set allowinsecureregistries
param to true
, I am getting the below error. Any help pls?
Build image failed: peer not authenticated
I am using google container registry for storing images and I use <credHelper>gcloud</credHelper>
as credential helper.
I am using google container registry for storing images and I use gcloud as credential helper.
I think you should be using the gcr
credential helper. Do you have multiple accounts? It could also be that you're logged into the wrong account.
If that doesn't fix things, could you try enabling Google HTTP logging?
It sounds like somehow the SSL certificate validation is failing and so jib is failing over to using HTTP. If you set the com.google.api.client.http.level=ALL
then it will include live auth values and you can then try replaying the shown curl
commands to find out more.
it used to work after I upgrade from 0.9.4 version to 0.9.7 in the morning and after an hour it started failing. Does it ring any bell?
@adorearun is this still happening? We figured out that, if Jib 0.9.7 can't connect to the registry for reasons like the registry not listening or temporarily down (while allowInsecureRegistries
is not set), Jib prints out the misleading error message you saw: https://github.com/GoogleContainerTools/jib/issues/767#issuecomment-409799872
@DidierSchonne same goes for you. If this is still happening (while allowInsecureRegistries
is not set), I believe what below can actually mean is that you just cannot establish normal connection to gcr.io
for whatever reasons.
Only secure connections are allowed, but tried to reach URL: http://gcr.io/v2/distroless/java/manifests/latest
@chanseokoh so I confirm that I'm not behind a proxy and I still have the same issue.
Could you please try with the newly-released 0.9.8? It includes some fixes that may apply here, including dealing with SSL issues.
If the problems are still occurring, please run with -DjibSerialize=true
(prevents performing operations in parallel) and enable the Google HTTP logging. Try with com.google.api.client.http.level=ALL
and if nothing obvious jumps out, try again with com.google.api.client.http.level=CONFIG
(To hide airy info) and attach the output here.
@briandealwis I tried upgrading to 0.9.8 and run using -DjibSerialize=true and got the below exception com.google.cloud.tools.jib.registry.InsecureRegistryException:
Failed to verify the server at https://us.gcr.io/v2/<projectID>/<imageID>/blobs/shaxxxxxx because only secure connections are allowed
at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.handleUnverifiableServerException(RegistryEndpointCaller.java:160)
at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling(RegistryEndpointCaller.java:153)
@adorearun could you run with -X
too and send the results? And the Google HTTP logging output would be helpful too.
@briandealwis 0.9.8 did not resolve the issue directly BUT if I enable allowInsecureRegistries
, it works, I well have my image pushed to my private registry. I guess that distroless is downloaded through http and then my image pushed through https.
I'm going to update this comment with logs to let you investigate why distroless is not downloaded without allowInsecureRegistries
enabled.
@briandealwis Please find the exception below after following the steps .
Aug 02, 2018 9:58:22 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST --------------
HEAD https://us.gcr.io/v2/projectID/imageid/blobs/sha256:0a4ee1511969276095d958e5f15313637cc310e731623223979b7ea662a8f167
Accept:
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: jib 0.9.8 jib-maven-plugin Google-HTTP-Java-Client/1.23.0 (gzip)
Aug 02, 2018 9:58:22 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -X HEAD -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'User-Agent: jib 0.9.8 jib-maven-plugin Google-HTTP-Java-Client/1.23.0 (gzip)' -- 'https://us.gcr.io/v2/projectID/imageid/blobs/sha256:0a4ee1511969276095d958e5f15313637cc310e731623223979b7ea662a8f167'
[DEBUG] TIMED Pushing BLOB digest: sha256:0a4ee1511969276095d958e5f15313637cc310e731623223979b7ea662a8f167, size: 25978 : 84.093 ms
[DEBUG] Pushing BLOB digest: sha256:0a4ee1511969276095d958e5f15313637cc310e731623223979b7ea662a8f167, size: 25978 : 84.093 ms
[DEBUG] TIMED Setting up to push layers : 271.796 ms
[DEBUG] Setting up to push layers : 271.796 ms
[INFO] Finalizing...
Aug 02, 2018 9:58:22 AM com.google.common.util.concurrent.AggregateFuture$RunningState handleException
SEVERE: Got more than one input Future failure. Logging failures after the first
com.google.cloud.tools.jib.registry.InsecureRegistryException: Failed to verify the server at https://us.gcr.io/v2/projectid/imageid/blobs/sha256:1607093a898cc241de8712e4361dcd907898fff35b945adca42db3963f3827b3 because only secure connections are allowed.
at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.handleUnverifiableServerException(RegistryEndpointCaller.java:160)
at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling(RegistryEndpointCaller.java:153)
@DidierSchonne and @adorearun what OS and Java runtime are you running? I wonder if your JRE's Certification Authority roots have been changed?
Could you please run the following three commands and report back on the output:
java -version
Assuming you have OpenSSL installed, let's see what certificate issues OpenSSL reports:
openssl s_client -showcerts -connect us.gcr.io:443 < /dev/null
And could you run the following to output the CA certs from your JRE keystore:
keytool -list -rfc -storepass changeit -keystore $JAVA_HOME/jre/lib/security/cacerts
The last part may need to change: if you're running a JRE by default, then it'll just be $JAVA_HOME/lib/security/cacerts
. We're looking to ensure there's a globalsignr2ca
certificate listed.
I see the following certificate chain when connecting with openssl
:
$ openssl s_client -showcerts -connect us.gcr.io:443 </dev/null
Yeah, we are on the right track, and it all makes sense now. The cause of the failure is that both @DidierSchonne @adorearun cannot verify the TLS certificate of gcr.io
at some point for some reason. That explains all the error messages and behaviors of Jib 0.9.7 and 0.9.8. There can be many reasons, but reasons can well be a problem in the local environment (e.g., you have an outdated or modified list of trusted root CA certificates, using some custom JDK, behind a proxy, etc.).
0.9.8 now allows connecting to an HTTPS server whose certificate cannot be verified if allowInsecureRegistries
is set (previously with 0.9.7, it could only try HTTP, which gcr.io
just doesn't listen on), so that's why 0.9.8 works. But still, for some unknown reason, you cannot verify the authenticity of gcr.io
from your machine. @adorearun seems to have the same issue, so I expect allowInsecureRegistries
with 0.9.8 may work.
$ java -version
openjdk version "10" 2018-03-20
OpenJDK Runtime Environment 18.3 (build 10+46)
OpenJDK 64-Bit Server VM 18.3 (build 10+46, mixed mode)
$ openssl s_client -showcerts -connect us.gcr.io:443 < /dev/null
@chanseokoh @briandealwis I appreciate all your help and inputs here , please find the output below version -
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
I don't have any certificates with me so here is the output for keytool - `keytool error: java.lang.Exception: Keystore file does not exist: /jre/lib/security/cacerts`
@DidierSchonne we've done some research and we have some evidence here and there to make us believe that your OpenJDK 10 cannot verify Google servers. For example, I downloaded OpenJDK 10 tar.gz, unpacked it, executed jshell
, and tried making connections to a few HTTPS servers:
jdk-10.0.2/bin$ ./jshell
...
jshell> new URL("https://oracle.com").openStream()
$1 ==> sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@2a32de6c
jshell> new URL("https://amazon.com").openStream()
$2 ==> sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@45018215
jshell> new URL("https://google.com").openStream()
| javax.net.ssl.SSLHandshakeException thrown: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
| at Alerts.getSSLException (Alerts.java:198)
| at SSLSocketImpl.fatal (SSLSocketImpl.java:1974)
| at Handshaker.fatalSE (Handshaker.java:345)
| at Handshaker.fatalSE (Handshaker.java:339)
| at ClientHandshaker.checkServerCerts (ClientHandshaker.java:1968)
| at ClientHandshaker.serverCertificate (ClientHandshaker.java:1777)
| at ClientHandshaker.processMessage (ClientHandshaker.java:264)
| at Handshaker.processLoop (Handshaker.java:1098)
| at Handshaker.processRecord (Handshaker.java:1026)
| at SSLSocketImpl.processInputRecord (SSLSocketImpl.java:1137)
| at SSLSocketImpl.readRecord (SSLSocketImpl.java:1074)
| at SSLSocketImpl.readRecord (SSLSocketImpl.java:973)
| at SSLSocketImpl.performInitialHandshake (SSLSocketImpl.java:1402)
| at SSLSocketImpl.startHandshake (SSLSocketImpl.java:1429)
| at SSLSocketImpl.startHandshake (SSLSocketImpl.java:1413)
| at HttpsClient.afterConnect (HttpsClient.java:567)
| at AbstractDelegateHttpsURLConnection.connect (AbstractDelegateHttpsURLConnection.java:185)
| at HttpURLConnection.getInputStream0 (HttpURLConnection.java:1581)
| at HttpURLConnection.getInputStream (HttpURLConnection.java:1509)
| at HttpsURLConnectionImpl.getInputStream (HttpsURLConnectionImpl.java:245)
| at URL.openStream (URL.java:1117)
| at (#3:1)
There seem to exist several related JDK bugs (e.g., https://bugs.openjdk.java.net/browse/JDK-8207255).
@DidierSchonne so I expect if you downgrade to Java 8 for example, you should be able to verify Google servers.
@adorearun I did notice you are not using OpenJDK 10, but I wonder if the build tool you are using is picking up some other JRE. mvn -v
would tell you which JRE it uses.
@chanseokoh @briandealwis yep we are not using OpenJDK please find the output
Maven home: /usr/local/Cellar/maven/3.5.4/libexec
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
@adorearun can you do
keytool -keystore /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/lib/security/cacerts -v -list | grep GlobalSign
to see if your JDK has GlobalSign CA certs? It will ask a password, and if you have never modified your JDK, it should be "changeit". For my JDK for example, I get
$ keytool -keystore ./cacerts -v -list | grep GlobalSign
Enter keystore password: changeit
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R3
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R3
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R5
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R5
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R4
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R4
Owner: CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE
Issuer: CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2
@DidierSchonne Probably the easiest thing to do is to use a tool like KeyStore Explorer and import either the GlobalSign Root CA - R2 certificate and the GTS certificates (Google Trust Services) from Google Trust Service's "sample PEM file" (see What roots should we trust for connecting to Google? in the Google Trust Service's FAQ). This is the same process required for using a registry with a self-signed certificate.
@DidierSchonne Probably the easiest thing to do is to use a tool like KeyStore Explorer and import
I find using the keytool
CLI more convenient here.
I tried importing the Google Trust Service's "sample PEM file". Adding it didn't help. (UPDATE: @briandealwis later told me that the following command (the -import
switch) imports only the first certificate in the pem
file.)
jdk-10.0.2/bin$ ./keytool -import -trustcacerts -alias gtsroots -cacerts -file ~/Downloads/roots.pem
Enter keystore password: changeit
Certificate already exists in keystore under alias <comodoaaaca [jdk]>
Do you still want to add it? [no]: yes
Certificate was added to keystore
jdk-10.0.2/bin$ ./jshell <<EOL
new URL("https://google.com").openStream()
EOL
...
| javax.net.ssl.SSLHandshakeException thrown: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
What worked for me was the GlobalSign Root R2 crt
file from here.
jdk-10.0.2/bin$ ./keytool -import -trustcacerts -alias gsr2 -cacerts -file ~/Downloads/GSR2.crt
Enter keystore password: changeit
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2
...
Trust this certificate? [no]: yes
Certificate was added to keystore
$ jdk-10.0.2/bin$ ./jshell <<EOL
new URL("https://google.com").openStream()
EOL
...
jshell> new URL("https://google.com").openStream()
$1 ==> sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@7995092a
So, if you have permission to modify your JDK, importing missing certificates could be one of possibly many other workarounds. (If you don't have the permission to modify your JDK, another option could be to install one on your home. It's also possible to leave the JDK trusted keystore intact and load a different keystore per individual Java application.)
@chanseokoh @briandealwis, thank you very much for your support, you found the issue. And you're right, with an 8 Java version it's ok. Next month Java 11 should be released, I hope that it will be fixed. BTW, Gitlab CI will build our app and use Jib to create the Docker image so we just have to be aware to use the right version.
Besides that I tried to add a <from>openjdk:10-jre</from>
or <from>openjdk:10</from>
tag into <configuration>
like in the doc and Jib crashes :
Unable to parse configuration of mojo com.google.cloud.tools:jib-maven-plugin:0.9.8:build for parameter from: Cannot find default setter in class com.google.cloud.tools.jib.maven.JibPluginConfiguration$FromConfiguration
Did I miss something, or did somehting wrong? Or should I open a new issue?
Thanks a lot again for your great responsiveness.
@chanseokoh @briandealwis please find the output for global signed certificate in my JDK
Enter keystore password: changeit
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R4
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R4
Owner: CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE
Issuer: CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R3
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R3
Owner: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R5
Issuer: CN=GlobalSign, O=GlobalSign, OU=GlobalSign ECC Root CA - R5
@DidierSchonne I opened a new issue for the "Cannot find default setter in class com.google.cloud.tools.jib.maven.JibPluginConfiguration$FromConfiguration" error: https://github.com/GoogleContainerTools/jib/issues/791
Using JDK 8 could be an easy workaround, but just keep in mind that there are other options or workarounds such as importing GlobalSign CA certs into your JDK's default trusted keystore or making your Java programs to load a different trusted keystore, if you do some research, but this might not worth your time. Lastly, it seems that OpenJDK-11 Early Access is also missing the root CA certs, so I also hope it will have them when officially released.
@adorearun then your issue seems different from @DidierSchonne's. Can you confirm that 0.9.8 with allowInsecureRegistries
works?
@chanseokoh @briandealwis Once I set the allowInsecureRegistries
param to true I am getting different exception below
Build image failed: Failed to authenticate with the registry because: peer not authenticated
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates (SSLSessionImpl.java:440)
at org.apache.http.conn.ssl.AbstractVerifier.verify (AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket (SSLSocketFactory.java:339)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection (DefaultClientConnectionOperator.java:123)
at org.apache.http.impl.conn.AbstractPoolEntry.open (AbstractPoolEntry.java:147)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open (AbstractPooledConnAdapter.java:108)
at org.apache.http.impl.client.DefaultRequestDirector.execute (DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:576)
at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:554)
at com.google.api.client.http.apache.ApacheHttpRequest.execute (ApacheHttpRequest.java:65)
at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:981)
at com.google.cloud.tools.jib.http.Connection.send (Connection.java:161)
at com.google.cloud.tools.jib.http.Connection.get (Connection.java:116)
at com.google.cloud.tools.jib.registry.RegistryAuthenticator.authenticate (RegistryAuthenticator.java:253)
at com.google.cloud.tools.jib.registry.RegistryAuthenticator.authenticatePush (RegistryAuthenticator.java:226)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call (AuthenticatePushStep.java:92)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call (AuthenticatePushStep.java:42)
at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly (CombinedFuture.java:181)
at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:57)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute (MoreExecutors.java:258)
at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute (CombinedFuture.java:112)
at com.google.common.util.concurrent.CombinedFuture$CombinedFutureRunningState.handleAllCompleted (CombinedFuture.java:75)
at com.google.common.util.concurrent.AggregateFuture$RunningState.processCompleted (AggregateFuture.java:261)
at com.google.common.util.concurrent.AggregateFuture$RunningState.decrementCountAndMaybeComplete (AggregateFuture.java:248)
at com.google.common.util.concurrent.AggregateFuture$RunningState.access$300 (AggregateFuture.java:94)
at com.google.common.util.concurrent.AggregateFuture$RunningState$1.run (AggregateFuture.java:151)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute (MoreExecutors.java:397)
at com.google.common.util.concurrent.AbstractFuture.executeListener (AbstractFuture.java:1016)
at com.google.common.util.concurrent.AbstractFuture.addListener (AbstractFuture.java:672)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener (AbstractFuture.java:107)
at com.google.common.util.concurrent.AggregateFuture$RunningState.init (AggregateFuture.java:144)
at com.google.common.util.concurrent.AggregateFuture$RunningState.access$100 (AggregateFuture.java:94)
at com.google.common.util.concurrent.AggregateFuture.init (AggregateFuture.java:91)
at com.google.common.util.concurrent.CombinedFuture.<init> (CombinedFuture.java:52)
at com.google.common.util.concurrent.Futures$FutureCombiner.call (Futures.java:993)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.<init> (AuthenticatePushStep.java:60)
at com.google.cloud.tools.jib.builder.steps.StepsRunner.runAuthenticatePushStep (StepsRunner.java:93)
at com.google.cloud.tools.jib.builder.BuildSteps.lambda$forBuildToDockerRegistry$0 (BuildSteps.java:84)
at com.google.cloud.tools.jib.builder.BuildSteps.run (BuildSteps.java:208)
at com.google.cloud.tools.jib.frontend.BuildStepsRunner.build (BuildStepsRunner.java:211)
at com.google.cloud.tools.jib.maven.BuildImageMojo.execute (BuildImageMojo.java:173)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
@adorearun that makes sense. For some unknown reason, you cannot still verify HTTPS servers.
(You may wonder why allowInsecureRegistries
did not bypass server verification failure; that's because we have an issue that there is another code path where allowInsecureRegistries
doesn't apply: https://github.com/GoogleContainerTools/jib/issues/720)
I'm currently clueless. Does this also happen when you use GCR for both the from image and the to image? If possible, can you test with other registries? I'm wondering if you cannot verify only Google registries.
@chanseokoh @briandealwis I believe both from and to will take from GCR .My configuration below
<configuration>
<from>
<image>openjdk:8</image>
</from>
<to>
<image>us.gcr.io/projectid/imagename</image>
<credHelper>gcloud</credHelper>
</to>
<allowInsecureRegistries>true</allowInsecureRegistries>
<container>
<jvmFlags>
<jvmFlag>-Dspring.profiles.active=gcpcloud</jvmFlag>
</jvmFlags>
<mainClass>com.XXX.mm.rd.XXX.XXX.myApplication</mainClass>
<args>
<arg>APPLICATION_ARGS</arg>
</args>
<ports>
</ports>
</container>
</configuration>
@adorearun no, openjdk:8
is from Docker Hub. I'd like to know if Docker Hub is OK and only GCR has this problem. Can you try pushing your image to Docker Hub? (If you have never had a Docker Hub account, it's easy to create. Setting up a private repo for pushing an image is also easy.)
Also, I see you are pushing to us.gcr.io
. What if you just have gcr.io/projectid/imagename
without us
?
@chanseokoh I have seen a strange behavior , I changed us.gcr.io to gcr.io in project 1 then tried jib build and it works. I have another project 2 , I changed us.gcr.io to gcr.io in project 2 then tried jib build and it failed.
Now I tried project 1 which was working earlier , is failing now. . very strange right.
@adorearun based on all of the logs and info you gave us, here is what I think. But before that, let's examine some of your logs:
Aug 02, 2018 9:58:22 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST --------------
HEAD https://us.gcr.io/v2/projectID/imageid/blobs/sha256:0a4ee1511969276095d958e5f15313637cc310e731623223979b7ea662a8f167
...
Authorization: <Not Logged>
...
handleException
SEVERE: Got more than one input Future failure. Logging failures after the first
com.google.cloud.tools.jib.registry.InsecureRegistryException: Failed to verify the server at https://us.gcr.io/v2/projectid/imageid/blobs/sha256:1607093a898cc241de8712e4361dcd907898fff35b945adca42db3963f3827b3 because only secure connections are allowed.
at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.handleUnverifiableServerException(RegistryEndpointCaller.java:160)
at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling(RegistryEndpointCaller.java:153)
The log was when allowInsecureRegistries
was not enabled. This operation log shows that Jib was trying to check if GCR already caches a particular image layer (a blob identified by 607093a898cc241de8712e4361dcd907898fff35b945adca42db3963f3827b3
). The thing is, this is well after Jib made a few successful interactions with us.gcr.io
. For example, it's sending the Authorization
credentials along the HEAD request, which must have been previously obtained from us.gcr.io
. This means, prior to this operation, you were able to verify certificates of us.gcr.io
. It worked.
You also said
It was strange for me because it used to work after I upgrade from 0.9.4 version to 0.9.7 in the morning and after an hour it started failing. Does it ring any bell?
and
I have seen a strange behavior , I changed us.gcr.io to gcr.io in project 1 then tried jib build and it works. I have another project 2 , I changed us.gcr.io to gcr.io in project 2 then tried jib build and it failed.
Now I tried project 1 which was working earlier , is failing now. . very strange right.
So I'm thinking your local environment is showing some unpredictable behavior regarding being able to verify SSL certificates of some servers including us.gcr.io
. Sometimes it works, but sometimes it doesn't.
Description of the issue: Error when pushing image to a private Gitlab Registry with Maven plugin 0.9.7
Expected behavior: An image is built and pushed to my private Gitlab Registry
Steps to reproduce:
settings.xml
in plain text to avoid additional issuessudo mvn compile -X com.google.cloud.tools:jib-maven-plugin:0.9.7:build
Environment:
jib-maven-plugin
Configuration:Log output:
Error Log:
``` $ sudo mvn compile -X com.google.cloud.tools:jib-maven-plugin:0.9.7:build Apache Maven 3.5.2 Maven home: /usr/share/maven Java version: 10.0.1, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-11-openjdk-amd64 Default locale: fr_FR, platform encoding: UTF-8 OS name: "linux", version: "4.15.0-29-generic", arch: "amd64", family: "unix" WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/maven/lib/guice.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [DEBUG] Created new class realm maven.api [DEBUG] Importing foreign packages into class realm maven.api [DEBUG] Imported: javax.annotation.* < plexus.core [DEBUG] Imported: javax.enterprise.inject.* < plexus.core [DEBUG] Imported: javax.enterprise.util.* < plexus.core [DEBUG] Imported: javax.inject.* < plexus.core [DEBUG] Imported: org.apache.maven.* < plexus.core [DEBUG] Imported: org.apache.maven.artifact < plexus.core [DEBUG] Imported: org.apache.maven.classrealm < plexus.core [DEBUG] Imported: org.apache.maven.cli < plexus.core [DEBUG] Imported: org.apache.maven.configuration < plexus.core [DEBUG] Imported: org.apache.maven.exception < plexus.core [DEBUG] Imported: org.apache.maven.execution < plexus.core [DEBUG] Imported: org.apache.maven.execution.scope < plexus.core [DEBUG] Imported: org.apache.maven.lifecycle < plexus.core [DEBUG] Imported: org.apache.maven.model < plexus.core [DEBUG] Imported: org.apache.maven.monitor < plexus.core [DEBUG] Imported: org.apache.maven.plugin < plexus.core [DEBUG] Imported: org.apache.maven.profiles < plexus.core [DEBUG] Imported: org.apache.maven.project < plexus.core [DEBUG] Imported: org.apache.maven.reporting < plexus.core [DEBUG] Imported: org.apache.maven.repository < plexus.core [DEBUG] Imported: org.apache.maven.rtinfo < plexus.core [DEBUG] Imported: org.apache.maven.settings < plexus.core [DEBUG] Imported: org.apache.maven.toolchain < plexus.core [DEBUG] Imported: org.apache.maven.usability < plexus.core [DEBUG] Imported: org.apache.maven.wagon.* < plexus.core [DEBUG] Imported: org.apache.maven.wagon.authentication < plexus.core [DEBUG] Imported: org.apache.maven.wagon.authorization < plexus.core [DEBUG] Imported: org.apache.maven.wagon.events < plexus.core [DEBUG] Imported: org.apache.maven.wagon.observers < plexus.core [DEBUG] Imported: org.apache.maven.wagon.proxy < plexus.core [DEBUG] Imported: org.apache.maven.wagon.repository < plexus.core [DEBUG] Imported: org.apache.maven.wagon.resource < plexus.core [DEBUG] Imported: org.codehaus.classworlds < plexus.core [DEBUG] Imported: org.codehaus.plexus.* < plexus.core [DEBUG] Imported: org.codehaus.plexus.classworlds < plexus.core [DEBUG] Imported: org.codehaus.plexus.component < plexus.core [DEBUG] Imported: org.codehaus.plexus.configuration < plexus.core [DEBUG] Imported: org.codehaus.plexus.container < plexus.core [DEBUG] Imported: org.codehaus.plexus.context < plexus.core [DEBUG] Imported: org.codehaus.plexus.lifecycle < plexus.core [DEBUG] Imported: org.codehaus.plexus.logging < plexus.core [DEBUG] Imported: org.codehaus.plexus.personality < plexus.core [DEBUG] Imported: org.codehaus.plexus.util.xml.Xpp3Dom < plexus.core [DEBUG] Imported: org.codehaus.plexus.util.xml.pull.XmlPullParser < plexus.core [DEBUG] Imported: org.codehaus.plexus.util.xml.pull.XmlPullParserException < plexus.core [DEBUG] Imported: org.codehaus.plexus.util.xml.pull.XmlSerializer < plexus.core [DEBUG] Imported: org.eclipse.aether.* < plexus.core [DEBUG] Imported: org.eclipse.aether.artifact < plexus.core [DEBUG] Imported: org.eclipse.aether.collection < plexus.core [DEBUG] Imported: org.eclipse.aether.deployment < plexus.core [DEBUG] Imported: org.eclipse.aether.graph < plexus.core [DEBUG] Imported: org.eclipse.aether.impl < plexus.core [DEBUG] Imported: org.eclipse.aether.installation < plexus.core [DEBUG] Imported: org.eclipse.aether.internal.impl < plexus.core [DEBUG] Imported: org.eclipse.aether.metadata < plexus.core [DEBUG] Imported: org.eclipse.aether.repository < plexus.core [DEBUG] Imported: org.eclipse.aether.resolution < plexus.core [DEBUG] Imported: org.eclipse.aether.spi < plexus.core [DEBUG] Imported: org.eclipse.aether.transfer < plexus.core [DEBUG] Imported: org.eclipse.aether.version < plexus.core [DEBUG] Imported: org.fusesource.jansi.* < plexus.core [DEBUG] Imported: org.slf4j.* < plexus.core [DEBUG] Imported: org.slf4j.helpers.* < plexus.core [DEBUG] Imported: org.slf4j.spi.* < plexus.core [DEBUG] Populating class realm maven.api [INFO] Error stacktraces are turned on. [DEBUG] Message scheme: color [DEBUG] Message styles: debug info warning error success failure strong mojo project [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml [DEBUG] Reading user settings from /root/.m2/settings.xml [DEBUG] Reading global toolchains from /usr/share/maven/conf/toolchains.xml [DEBUG] Reading user toolchains from /root/.m2/toolchains.xml [DEBUG] Using local repository at /root/.m2/repository [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /root/.m2/repository [INFO] Scanning for projects... [DEBUG] Extension realms for project com.mydomain.sandbox:slow-response:jar:0.0.1-SNAPSHOT: (none) [DEBUG] Looking up lifecycle mappings for packaging jar from ClassRealm[plexus.core, parent: null] [DEBUG] Extension realms for project org.springframework.boot:spring-boot-starter-parent:pom:2.0.2.RELEASE: (none) [DEBUG] Looking up lifecycle mappings for packaging pom from ClassRealm[plexus.core, parent: null] [DEBUG] Extension realms for project org.springframework.boot:spring-boot-dependencies:pom:2.0.2.RELEASE: (none) [DEBUG] Looking up lifecycle mappings for packaging pom from ClassRealm[plexus.core, parent: null] [DEBUG] === REACTOR BUILD PLAN ================================================ [DEBUG] Project: com.mydomain.sandbox:slow-response:jar:0.0.1-SNAPSHOT [DEBUG] Tasks: [compile, com.google.cloud.tools:jib-maven-plugin:0.9.7:build] [DEBUG] Style: Regular [DEBUG] ======================================================================= [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building slow-response 0.0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [DEBUG] Lifecycle default -> [validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy] [DEBUG] Lifecycle clean -> [pre-clean, clean, post-clean] [DEBUG] Lifecycle site -> [pre-site, site, post-site, site-deploy] [DEBUG] Lifecycle default -> [validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy] [DEBUG] Lifecycle clean -> [pre-clean, clean, post-clean] [DEBUG] Lifecycle site -> [pre-site, site, post-site, site-deploy] [DEBUG] Lifecycle default -> [validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy] [DEBUG] Lifecycle clean -> [pre-clean, clean, post-clean] [DEBUG] Lifecycle site -> [pre-site, site, post-site, site-deploy] [DEBUG] Lifecycle default -> [validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy] [DEBUG] Lifecycle clean -> [pre-clean, clean, post-clean] [DEBUG] Lifecycle site -> [pre-site, site, post-site, site-deploy] [DEBUG] === PROJECT BUILD PLAN ================================================ [DEBUG] Project: com.mydomain.sandbox:slow-response:0.0.1-SNAPSHOT [DEBUG] Dependencies (collect): [] [DEBUG] Dependencies (resolve): [compile, runtime+system] [DEBUG] Repositories (dependencies): [central (https://repo.maven.apache.org/maven2, default, releases)] [DEBUG] Repositories (plugins) : [central (https://repo.maven.apache.org/maven2, default, releases)] [DEBUG] ----------------------------------------------------------------------- [DEBUG] Goal: org.apache.maven.plugins:maven-resources-plugin:3.0.1:resources (default-resources) [DEBUG] Style: Regular [DEBUG] Configuration:Thank you for your help.