Failed to use image from private Docker repository

[cwansart]

Hello,

we are running a Nexus Repository Manager which has an http proxy (not https) configured to reach Docker Hub.

In a new test using testcontainers' Oracle database image, it fails to get it from the proxy repository. I already created a testcontainers.properties with oracle.container.image=10.20.30.40:18081/wnameless/oracle-xe-11g but it still tries to access the docker index and repository.

10:47:14.601 [dockerjava-netty-1-12] (MDC: ) ERROR com.github.dockerjava.core.async.ResultCallbackTemplate - Error during callback com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://10.22.30.40:18081/v2/: http: server gave HTTP response to HTTPS client"}

Addition: According to https://stackoverflow.com/questions/38695515/can-not-pull-push-images-after-update-docker-to-1-12 I have to add "Insecure Repositories" which I have done on my server as well as on my local machine, running macOS. For reason it still tries to queries the Nexus via https.

I manually fetched it as the Jenkins user from our internal repository and it worked. What am I missing?

[kiview]

Hi @cwansart,

You need to add insecure-registries, not repositores, did you configure it correctly? Can you share your daemon.json?

It also might be a bug in docker-java, since the official docs indicate, that Docker will fallback to http after trying https:

First, try using HTTPS.

• If HTTPS is available but the certificate is invalid, ignore the error about the certificate.
• If HTTPS is not available, fall back to HTTP.

Also in your above example, I see 2 different IPs, is this to be extected? 10.20.30.40:18081 and 10.22.30.40:18081.

[cwansart]

Hi @kiview

the insecure-registries entries are already in the daemon.json. On the same machines we are using them via Jenkins in Jenkinsfiles without any issue.

I was already thinking about an issue in docker-java. I tried to add a docker-java.properties file containing the private registry, as noted in https://github.com/docker-java/docker-java#properties-docker-javaproperties, but it failed.

I am currently working on it getting Nexus running with a self-signed cert. The biggest issue I see is, when there are components like Java that gets updated. Then someone needs to install the certificate again.

Regarding the different IPs: It is a mistake. Both should be the same, they are obfuscated and not the real ones.

[kiview]

Can you please run docker pull 10.20.30.40:18081/wnameless/oracle-xe-11g manually on the machine and show the output?

Or was this exactly what you did here:

I manually fetched it as the Jenkins user from our internal repository and it worked.

Where are you mainly trying to get it working now? On your local machine or server?

[cwansart]

It now works on the server. For some reason anonymous pull does not work, but the certificates gets accepted now, when running docker pull by myself. The nexts stage is to let testcontainers use the registry with ssl.

Or was this exactly what you did here:

This was the pull you mentioned. It worked for the http registry, but testcontainers failed to use it. Now let's see if it fails for the https registry as well...

Where are you mainly trying to get it working now? On your local machine or server?

local machine would be nice, but the main target is the server.

Edit: It works now, but not on my local macOS based machine. Testcontainers fails with the error message

Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://10.220.60.194:18082/v2/: x509: certificate signed by unknown authority"}

But I can pull it without any issues when running

docker pull 10.20.30.40:18082/wnameless/oracle-xe-11g

Note: I started the https docker hub proxy on the port 18082. So for some reason testcontainers does not recognize the certificate added to the keystone in macOS.

Edit2: I made a mistake by using the IP address in the testcontainers.properties instead of the hostname "nexus" which is in my hosts file. For some reason I get this message now, after changing the ip to the hostname:

Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://nexus:18082/v2/: dial tcp: lookup nexus on 10.0.2.3:53: server misbehaving"}

The IP address does not fit, it should be 10.20.30.40, but it is 10.0.2.3 on port 53 for some reason...

[cwansart]

I am extremely confused now... On my local machine it fails with the error:

com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://nexus:18082/v2/: dial tcp: lookup nexus on 10.0.2.3:53: server misbehaving"}

On my Jenkins it fails by trying to reach Docker Hub. The code is the same. I added testcontainers.properties to src/main/resources. It has the following contents:

oracle.container.image=nexus:18082/wnameless/oracle-xe-11g

On my local machine it tries to get the Oracle container from nexus, on Jenkins it tries to get it from Docker Hub. The error message there is:

com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}

What am I missing?

[kiview]

Sorry, this seems to be a quite specific problem for your infrastructure, according to this post, it could be related to your DNS configuration: https://stackoverflow.com/questions/28301151/docker-pull-error

[rnorth]

Since this ticket was raised we've made significant improvements to how we tackle image pull authentication.

I'm going to assume, based on the age of the ticket and no further comments, that this can be safely closed.