V2_REGISTRY returns lowercase header and fails

[lerwys]

In migrator.sh line 811

https://github.com/docker/migrator/blob/5e7ef10ace96e34d33e8870173f230e738a292cf/migrator.sh#L811

The "if" expects "Docker-Distribution-Api-Version: registry/2.0", but it fails with my current registry 2.6.2, because the returned header is the following:

...
docker-distribution-api-version: registry/2.0
...

Am I using some wrong version of the migrator/registry or is the check maybe outdated?

Thanks!

Lucas

[mbentley]

I am running registry github.com/docker/distribution v2.6.2 and when I curl, I get the appropriate case:

# curl -v http://localhost:5001/v2/
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5001 (#0)
> GET /v2/ HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Mon, 21 May 2018 18:34:19 GMT
<
* Connection #0 to host localhost left intact

Are you running something like nginx in front and it is returning the header that is injected in? If so, it sounds like it is just inserted incorrectly in the configuration of the front end load balancer. Here in the docs, it mentions the requirement for this header: https://github.com/docker/distribution/blob/master/docs/spec/api.md

I am not sure if the docker client requires that to be exact with the same case but I was just following the documented header. RFC 2616 does mention that headers are case insensitive so I suppose this could be change to ignore case.

[lerwys]

Humm.... you're right, I think.

The original issue I was having was the following. I issued the following command

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc/docker/certs.d:/etc/docker/certs.d:ro -e V1_REGISTRY=docker.io -e V2_REGISTRY=dockerregistry.lnls-sirius.com.br -e DOCKER_HUB_ORG=lnlsdig -e V1_REPO_FILTER="epics-ioc" docker/migrator 2>&1 | tee migration.log

It correctly pulled all of the images but it couldn't push them to my registry:

[ERROR] v2 registry (dockerregistry.lnls-sirius.com.br) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

I can reach the [dockerregistry.lnls-sirius.com.br]()

nslookup dockerregistry.lnls-sirius.com.br
Server:     10.2.128.31
Address:    10.2.128.31#53

Name:   dockerregistry.lnls-sirius.com.br
Address: 10.2.128.31

But the command "curl -v" returns this:

dig@dig-cpu-test-ioc:~/Repos/migrator$ curl -v https://dockerregistry.lnls-sirius.com.br/v2

*   Trying 10.2.128.31...
* TCP_NODELAY set
* Connected to dockerregistry.lnls-sirius.com.br (10.2.128.31) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=BR; ST=Sao Paulo; L=Campinas; O=CNPEM; OU=LNLS; CN=dockerregistry.lnls-sirius.com.br
*  start date: Apr  4 20:14:05 2018 GMT
*  expire date: Apr  4 20:14:05 2019 GMT
*  common name: dockerregistry.lnls-sirius.com.br (matched)
*  issuer: C=BR; ST=Sao Paulo; L=Campinas; O=CNPEM; OU=LNLS; CN=dockerregistry.lnls-sirius.com.br
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557d30c6adc0)
> GET /v2 HTTP/1.1
> Host: dockerregistry.lnls-sirius.com.br
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 301 
< docker-distribution-api-version: registry/2.0
< location: /v2/
< content-type: text/html; charset=utf-8
< content-length: 39
< date: Tue, 22 May 2018 11:34:54 GMT
< 
<a href="/v2/">Moved Permanently</a>.

* Curl_http_done: called premature == 0
* Connection #0 to host dockerregistry.lnls-sirius.com.br left intact

Maybe this is related to my self-signed certificate?

[mbentley]

Do you know how your v2 is deployed? It wouldn't be related to the certificates as headers aren't modified by your certs. Either way, I will make the grep be case insensitive to match RFC 2616 stating that headers are to be case insensitive.

[lerwys]

Ok,

I use a compose file like this:

version: '3'

services:
  registry:
    restart: always
    image: registry:2.6.2
    ports:
      - "443:5000/tcp"
    container_name: registry
    environment:
      REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
      REGISTRY_HTTP_TLS_KEY: /certs/domain.key
      REGISTRY_HTTP_ADDR: 0.0.0.0:5000
    volumes:
      - ./foreign/docker-registry-certs/certs/data:/var/lib/registry
      - ./foreign/docker-registry-certs/certs:/certs
      - ./foreign/docker-registry-certs/certs/auth:/auth

[lerwys]

So, even with the fix I can't get the migrator to work with my registry.

I can push/pull images to/from my registry:

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ docker push dockerregistry.lnls-sirius.com.br/lnlsdig/aravis-epics-ioc
The push refers to repository [dockerregistry.lnls-sirius.com.br/lnlsdig/aravis-epics-ioc]
29150f15e24c: Pushed 
bd1c16de2083: Pushed 
081bec28b1b9: Pushed 
d280cd00179a: Pushed 
0146196b0ec2: Pushed 
9778892089d8: Pushed 
5d2ed173cb4b: Pushed 
69c21860a250: Pushed 
a75caa09eb1f: Mounted from basler-aca1300-75gm-epics-ioc 
latest: digest: sha256:1786ef25ad165432879b76067535552fafd95a563f863c074e7a01c02b4c3fd2 size: 2224

But the error is still happening:

[ERROR] v2 registry (dockerregistry.lnls-sirius.com.br) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

Any ideias on how to debug this?

Thanks for the support so far!

Lucas

[mbentley]

Ah, with self signed certs, you have two options:

Use the env var for USE_INSECURE_CURL=true

-or-

You can pass the CA certificate from your v2 since /etc/docker/certs.d is a volume that is passed into the migrator container. So if you have /etc/docker/certs.d/$V2_REGISTRY/ca.crt on your host you're running it from, it will work out of the box as there is a check here that will add the CA cert to the curl command so that it verifies.

[lerwys]

Hum, I tried with USE_INSECURE_CURL=true:

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc/docker/certs.d:/etc/docker/certs.d:ro -e V1_REGISTRY=docker.io -e USE_INSECURE_CURL=true -e V2_REGISTRY=dockerregistry.lnls-sirius.com.br -e DOCKER_HUB_ORG=lnlsdig -e V1_REPO_FILTER="epics-ioc:*base-3.15*" docker/migrator 2>&1 | tee migration.log

Without luck. I must be missing something obvious, I think...

[mbentley]

Let's do some tests using manual commands. Can you provide the output of this?:

curl --insecure -Is https://dockerregistry.lnls-sirius.com.br/v2/ | grep -i ^'Docker-Distribution-Api-Version: registry/2.0'

Like, here is mine:

# curl --insecure -Is https://registry-v2-noauth.core.dckrindy.io/v2/ | grep -i ^'Docker-Distribution-Api-Version: registry/2.0'
Docker-Distribution-Api-Version: registry/2.0

And then if you do have a file at /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt on the host, does this work?

curl --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/ | grep -i ^'Docker-Distribution-Api-Version: registry/2.0'

[lerwys]

Sure, here it is:

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ curl --insecure -Is https://dockerregistry.lnls-sirius.com.br/v2/ | grep -i ^'Docker-Distribution-Api-Version: registry/2.0'
docker-distribution-api-version: registry/2.0

The following doesn't output anything

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ curl --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/

maybe because /etc/docker has permissions 700?

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ sudo ls -la /etc/docker/
total 24
drwx------   3 root root  4096 May  4 14:12 .
drwxr-xr-x 134 root root 12288 May 21 08:30 ..
drwxr-xr-x   4 root root  4096 May 22 13:06 certs.d
-rw-------   1 root root   244 Nov 17  2017 key.json

[mbentley]

Inside the container, it would be running as root so that would be ok but for this test, can you run it as root? Also, try the same and get rid of the pipe to grep to get the full output.

[lerwys]

here it is:

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ sudo curl --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/
HTTP/2 200 
content-type: application/json; charset=utf-8
docker-distribution-api-version: registry/2.0
x-content-type-options: nosniff
content-length: 2
date: Tue, 22 May 2018 17:55:57 GMT

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ curl --insecure -Is https://dockerregistry.lnls-sirius.com.br/v2/
HTTP/2 200 
content-type: application/json; charset=utf-8
docker-distribution-api-version: registry/2.0
x-content-type-options: nosniff
content-length: 2
date: Tue, 22 May 2018 17:57:04 GMT

[mbentley]

Awesome; thanks for that. What happens when you combine the two?

curl --insecure --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/

Can you also make sure that the digest for you image matches the very latest?:

# docker images --digests  --filter reference=docker/migrator
REPOSITORY          TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
docker/migrator     latest              sha256:f07a83ad6edb9c143f71bf9857faa2a7ffad5d4820d5251fe2b96afbe86a0bf9   d08b09670855        4 hours ago         119MB

[lerwys]

Sure, no problem.

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ curl --insecure --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/
HTTP/2 200 
content-type: application/json; charset=utf-8
docker-distribution-api-version: registry/2.0
x-content-type-options: nosniff
content-length: 2
date: Tue, 22 May 2018 18:12:19 GMT

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ docker images --digests  --filter reference=docker/migrator
REPOSITORY          TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
docker/migrator     latest              sha256:f07a83ad6edb9c143f71bf9857faa2a7ffad5d4820d5251fe2b96afbe86a0bf9   d08b09670855        4 hours ago         119MB

[mbentley]

When you used USE_INSECURE_CURL, what was the error?

[lerwys]

When I used USE_INSECURE_CURL=true as -e USE_INSECURE_CURL=true in docker run I got this:

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc/docker/certs.d:/etc/docker/certs.d:ro -e V1_REGISTRY=docker.io -e USE_INSECURE_CURL=true -e V2_REGISTRY=dockerregistry.lnls-sirius.com.br -e DOCKER_HUB_ORG=lnlsdig -e V1_REPO_FILTER="epics-ioc:*base-3.15*" docker/migrator 2>&1 | tee migration.log

...
[OK] Successfully retagged docker.io/lnlsdig/aravis-epics-ioc:latest to dockerregistry.lnls-sirius.com.br/lnlsdig/aravis-epics-ioc:latest

[OK] Successfully retagged all images

[ERROR] v2 registry (dockerregistry.lnls-sirius.com.br) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

[mbentley]

OK, instead of trying from the host only, let's try from the container itself by starting bash:

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc/docker/certs.d:/etc/docker/certs.d:ro docker/migrator bash

Then once inside, let's run the curl commands again:

curl --insecure -Is https://dockerregistry.lnls-sirius.com.br/v2/

curl --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/

curl --insecure --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/

[lerwys]

Hum... Nothing appears

bash-4.3# curl --insecure -Is https://dockerregistry.lnls-sirius.com.br/v2/
bash-4.3# curl --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/
bash-4.3# curl --insecure --cacert /etc/docker/certs.d/dockerregistry.lnls-sirius.com.br/ca.crt -Is https://dockerregistry.lnls-sirius.com.br/v2/

So, from inside the docker it can't resolve the name dockerregistry.lnls-sirius.com.br, it seems

[mbentley]

Hmm, try with nslookup dockerregistry.lnls-sirius.com.br to see if it resolves at all. Do you have a proxy server that you have to go through?

If it is something where it is only a case of the DNS not resolving in the container, you can add --add-host foo.example.com:1.2.3.4 to the docker run command, replacing foo.example.com with dockerregistry.lnls-sirius.com.br and 1.2.3.4 with the IP that it resolves to.

[lerwys]

My DNS server is running locally, on the same machine as the registry, as is located at address 10.2.128.31. I did the following:

dig@dig-cpu-test-ioc:~/Repos/docker-registry-composed$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc/docker/certs.d:/etc/docker/certs.d:ro --dns=10.2.128.31  docker/migrator bash
bash-4.3# nslookup dockerregistry.lnls-sirius.com.br 10.2.128.31
Server:    10.2.128.31
Address 1: 10.2.128.31

nslookup: can't resolve 'dockerregistry.lnls-sirius.com.br': Try again

If I do the same command outside docker it works:

dig@dig-cpu-test-ioc:~/Repos/migrator$ nslookup dockerregistry.lnls-sirius.com.br 10.2.128.31
Server:     10.2.128.31
Address:    10.2.128.31#53

Name:   dockerregistry.lnls-sirius.com.br
Address: 10.2.128.31

[mbentley]

Sounds like your DNS server isn't allowing queries from the Docker IP range (my best guess; I had the same issue running my own bind server). Quick workaround would be to add this to your docker run:

--add-host dockerregistry.lnls-sirius.com.br:10.2.128.31 and then it should work; nslookup won't be it should be dropped in the /etc/hosts file.

[lerwys]

Oh.... Of course, how could I forget that? I only allow some other ranges, such as 10.2.128.0/24, and a couple others...

Everything is working now.

Thanks a lot for your help!

[mbentley]

Awesome, great to hear!