Following instructions in readme.md for test/dev/default registry not working

ForbiddenEra commented 9 years ago

Hi,

EDIT / TL:DR / ACTUAL ISSUE: Mostly figured it out except that I can't seem to use my own CA even when importing it 'properly' and mainly docs aren't clear that you are required to setup your own auth & ssl for the private registry to work in its most basic form. Also, error messages provided by the registry are very misleading (eg. see below where it tells me to visit my registry to activate a user it apparently accepted)

I'm trying to use the registry image out of the box with default settings (which says is the dev flavor, which is based off local storage, so it should just work..)

My network setup is a little different but I don't think that is affecting things (I've tried with default networking and same issue)

Whether I go (my normal way):

docker pull registry
docker run --name="registry0" --hostname="registry0" --net=none registry
sudo pipework docker0 registry0 10.0.0.5/24@10.0.0.1 // essentially assigns a local IP to my bridge
sudo weave attach 10.254.0.5/24 registry0 // adds a second network to my docker, which my other nodes can access regardless of where they are

or I just do:

docker run --name="registry0" --hostname="registry0" -p 5000:5000 registry
sudo weave attach 10.254.0.100/24 // again allows me to access it from another node

I get responses like this "logging in", I should note that I'm aware that the FAQ says standalone registry doesn't include user account control, but, I can't get it to work regardless. The FAQ says I can use an "nginx or Apache frontend with basic auth enabled" but doesn't says it's required to make it work at all..? Though, I'm thinking it is - if so this should be more clear!

I will try, but look below - this is very misleading..

core@core0 ~ $ docker login 10.254.0.100:5000
Username (shaped): 
Account created. Please see the documentation of the registry http://10.254.0.100:5000/v1/ for instructions how to activate it.
core@core0 ~ $ curl http://10.254.0.100:5000/v1/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>
core@core0 ~ $ 
core@core0 ~ $ docker login 10.254.0.100:5000
Username (shaped): 
Account created. Please see the documentation of the registry http://10.254.0.100:5000/v1/ for instructions how to activate it.
core@core0 ~ $ docker login 10.254.0.100:5000
Username (shaped): testing
Password: 
Email (jai@xxx.ca): test@xxx.ca
Account created. Please see the documentation of the registry http://10.254.0.100:5000/v1/ for instructions how to activate it.

Notice that it says that the account is created and gives me a link for instructions on how to activate it..?! what..?

or trying to push an image:

core@core0 ~ $ docker push shaped/haproxy
The push refers to a repository [shaped/haproxy] (len: 1)
Sending image list

Please login prior to push:
Username: shaped
Password: 
Email: shaped@shaped.ca
FATA[0006] Error response from daemon: Wrong login/password, please try again

And, of course, logs from the registry container:

core@core5 ~ $ docker run --name="registry0" --hostname="registry0" -p 5000:5000 registry
[2015-02-26 09:50:19 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-02-26 09:50:19 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-02-26 09:50:19 +0000] [1] [INFO] Using worker: gevent
[2015-02-26 09:50:19 +0000] [14] [INFO] Booting worker with pid: 14
[2015-02-26 09:50:19 +0000] [15] [INFO] Booting worker with pid: 15
[2015-02-26 09:50:19 +0000] [16] [INFO] Booting worker with pid: 16
[2015-02-26 09:50:19 +0000] [17] [INFO] Booting worker with pid: 17
26/Feb/2015:09:50:23 +0000 WARNING: Cache storage disabled!
26/Feb/2015:09:50:23 +0000 WARNING: LRU cache disabled!
26/Feb/2015:09:50:23 +0000 WARNING: Cache storage disabled!
26/Feb/2015:09:50:23 +0000 WARNING: Cache storage disabled!
26/Feb/2015:09:50:23 +0000 WARNING: LRU cache disabled!
26/Feb/2015:09:50:23 +0000 WARNING: Cache storage disabled!
26/Feb/2015:09:50:23 +0000 WARNING: LRU cache disabled!
26/Feb/2015:09:50:23 +0000 WARNING: LRU cache disabled!
26/Feb/2015:09:50:23 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
26/Feb/2015:09:50:23 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
26/Feb/2015:09:50:23 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
26/Feb/2015:09:50:23 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
26/Feb/2015:09:50:25 +0000 WARNING: Another process is creating the search database
26/Feb/2015:09:50:25 +0000 WARNING: Another process is creating the search database
26/Feb/2015:09:50:25 +0000 WARNING: Another process is creating the search database
10.254.0.200 - - [26/Feb/2015:09:52:14 +0000] "GET /v2/ HTTP/1.1" 404 233 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:52:14 +0000] "GET /v1/_ping HTTP/1.1" 200 1517 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:52:14 +0000] "POST /v1/users/ HTTP/1.1" 201 14 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:52:22 +0000] "GET /v1/ HTTP/1.1" 404 233 "-" "curl/7.30.0"
104.236.123.80 - - [26/Feb/2015:09:53:54 +0000] "GET /v1 HTTP/1.1" 404 233 "-" "curl/7.30.0"
104.236.123.80 - - [26/Feb/2015:09:53:58 +0000] "GET /v1 HTTP/1.1" 404 233 "-" "curl/7.30.0"
10.254.0.200 - - [26/Feb/2015:09:54:54 +0000] "GET /v2/ HTTP/1.1" 404 233 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:54:54 +0000] "GET /v1/_ping HTTP/1.1" 200 1517 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:54:54 +0000] "POST /v1/users/ HTTP/1.1" 201 14 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:55:04 +0000] "GET /v2/ HTTP/1.1" 404 233 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:55:04 +0000] "GET /v1/_ping HTTP/1.1" 200 1517 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:55:04 +0000] "POST /v1/users/ HTTP/1.1" 201 14 "-" "Go 1.1 package http"
10.254.0.200 - - [26/Feb/2015:09:57:38 +0000] "GET /v1/ HTTP/1.1" 404 233 "-" "curl/7.30.0"

Using image id c55308716b36 which is latest?

core@core0 ~ $ docker pull registry
511136ea3c5a: Already exists 
27d47432a69b: Already exists 
5f92234dcf1e: Already exists 
51a9c7c1f8bb: Already exists 
5ba9dab47459: Already exists 
30e25c7b70df: Already exists 
ed34dec80489: Already exists 
0e7a483810f6: Already exists 
eaebc036889a: Already exists 
8ec695ba9240: Already exists 
ecc59b06f5b7: Already exists 
63ad05f3af00: Already exists 
f054bc98768f: Already exists 
214c09aed08b: Already exists 
c55308716b36: Already exists 
registry:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Status: Image is up to date for registry:latest

ForbiddenEra commented 9 years ago

With nginx handling http_auth, I can login:

core@core0 ~ $ docker login 10.254.0.100:5000
Username (testing): shaped
Password: 
Email (test@shaped.ca): 
Login Succeeded

but I can't push an image:

core@core0 ~ $ docker tag shaped/haproxy0 10.254.0.100:5000/haproxy0       
core@core0 ~ $ docker push 10.254.0.100:5000/haproxy0 
The push refers to a repository [10.254.0.100:5000/haproxy0] (len: 1)
Sending image list
Pushing repository 10.254.0.100:5000/haproxy0 (1 tags)
511136ea3c5a: Pushing 
FATA[0000] HTTP code 401, Docker will not send auth headers over HTTP.

Which might sort of vaguely relate to #936 and referenced in #541 ..

So really - docs are not clear!!

ForbiddenEra commented 9 years ago

Implementing SSL leaves me about here:

I've added my CA to the machine and run update-ca-certificates

Then:

core@core0 /etc/ssl/certs $ docker login docker-registry:5000
Username: shaped
Password: 
Email: 
FATA[0003] Error response from daemon: v1 ping attempt failed with error: Get https://docker-registry:5000/v1/_ping: x509: certificate signed by unknown authority. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry docker-registry:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/docker-registry:5000/ca.crt

So, I take it's advice:

core@core0 /etc $ docker login docker-registry:5000
Username: shaped
Password: 
Email: 
FATA[0002] Error response from daemon: Server Error: Post https://docker-registry:5000/v1/users/: x509: certificate signed by unknown authority

....hmm

ForbiddenEra commented 9 years ago

I was able to get it to work but only by adding --insecure-registry="docker-registry:5000" to my docker start up line:

core@core0 /etc $ docker login docker-registry:5000
Username: shaped
Password: 
Email: 
Login Succeeded
core@core0 /etc $ docker tag shaped/haproxy0 docker-registry:5000/haproxy0
core@core0 /etc $ docker push docker-registry:5000/haproxy0 
The push refers to a repository [docker-registry:5000/haproxy0] (len: 1)
Sending image list
Pushing repository docker-registry:5000/haproxy0 (1 tags)
511136ea3c5a: Image successfully pushed 
53f858aaaf03: Image successfully pushed 
837339b91538: Image successfully pushed 
615c102e2290: Image successfully pushed 
b39b81afc8ca: Image successfully pushed 
8254ff58b098: Image successfully pushed 
ec5f59360a64: Image successfully pushed 
2ce4ac388730: Image successfully pushed 
2eccda511755: Image successfully pushed 
5a14c1498ff4: Image successfully pushed 
8ffd698b4b9a: Image successfully pushed 
c9950e27e2bf: Image successfully pushed 
f5489e95a03b: Image successfully pushed 
13e9704168f6: Image successfully pushed 
d329e079a86b: Image successfully pushed 
9675842043c7: Image successfully pushed 
949a55b1c715: Image successfully pushed 
9205a67b7f7d: Image successfully pushed 
70bee8e8629f: Image successfully pushed 
78934e85029e: Image successfully pushed 
Pushing tag for rev [78934e85029e] on {https://docker-registry:5000/v1/repositories/haproxy0/tags/latest}

My cert generation:

root@registry-gateway:~# openssl genrsa -out devdockerCA.key 2048
Generating RSA private key, 2048 bit long modulus
.......+++
.....................................................................................................................................................+++
e is 65537 (0x10001)
root@registry-gateway:~# openssl req -x509 -new -nodes -key devdockerCA.key -days 10000 -out devdockerCA.crt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:AB
Locality Name (eg, city) []:Calgary
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Shaped
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:shaped.ca
Email Address []:jai@shaped.ca
root@registry-gateway:~# openssl genrsa -out docker-registry.key 2048
Generating RSA private key, 2048 bit long modulus
........+++
.............+++
e is 65537 (0x10001)
root@registry-gateway:~# openssl req -new -key docker-registry.key -out docker-registry.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:AB
Locality Name (eg, city) []:Calgary
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Shaped
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:docker-registry
Email Address []:jai@shaped.ca

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
root@registry-gateway:~# openssl x509 -req -in docker-registry.csr -CA devdockerCA.crt -CAkey devdockerCA.key -CAcreateserial -out docker-registry.crt -days 10000
Signature ok
subject=/C=CA/ST=AB/L=Calgary/O=Shaped/CN=docker-registry/emailAddress=jai@shaped.ca
Getting CA Private Key
root@registry-gateway:~#

added devdockerCA.crt as /etc/ssl/certs/docker-dev-crt.pem and ran update-ca-certs which found it and added it and also to /etc/docker/certs.d/docker-registry:5000/ca.crt

The FAQ/docs should really say that an auth proxy & ssl is required even to be able to push/pull an image from a private registry - and there's NO other way..?

Still would rather have it working properly without --insecure-registry ... ideas?

dmp42 commented 9 years ago

Ok, there is a lot here.

Let's start with basic stuff:

docker run -p 5000:5000 registry should give you a running registry WITHOUT any authentication required
if you get anything else (like 401 or 403) then you are not talking to your registry, but to another http service
auth is NOT required to have a registry work
SSL is NOT required either (though no SSL means --insecure-registry)

Can you confirm that you got that step ok?

After that, adding SSL in the mix should go:

nginx configuration from here: https://github.com/docker/docker-registry/tree/master/contrib/nginx should get you started (disable the authentication to start with)
if your cert is self-signed, instructions here should get your docker host working by trusting your certificate: https://docs.docker.com/reference/commandline/cli/#insecure-registries

Once you get that second step ok, you can enable authentication back.

Let me know where you are now, and/or reach out on irc #docker-distribution so we can figure this out.

ForbiddenEra commented 9 years ago

Hi,

As I said above, running the basic registry directly from the repo would not give me anything usable. I am 100% not talking to another service - you can see clearly that when I access the repo (with no nginx in front) that the log for the registry. The only 401 I got was docker refusing to send auth headers over http - weird that it's a 401 and not a docker error? I thought that was part of the docker daemon.

As for adding SSL - I have no problem with that except docker isn't recognizing my CA - I have placed it in the correct location and it does pick it up (see logs) but still gives error.

However I will, for the sake of completion, start over..

And that makes me wonder -- I think maybe I forgot to tag my image for the repo before I pushed it the first time (resulting in that error?) because - now it's working without nginx/ssl / out of the box...?

I swear, I try to test anything and everything before opening an issue on github - and most of the time I finally get to that point, I find out I missed something small...? Thanks for your feedback though.

core@core0 ~ $ docker tag shaped/haproxy0 107.191.40.91:5000/haproxy0 
core@core0 ~ $ docker push 107.191.40.91:5000/haproxy0
The push refers to a repository [107.191.40.91:5000/haproxy0] (len: 1)
Sending image list
Pushing repository 107.191.40.91:5000/haproxy0 (1 tags)
511136ea3c5a: Image successfully pushed 
53f858aaaf03: Image successfully pushed 
837339b91538: Image successfully pushed 
615c102e2290: Image successfully pushed 
b39b81afc8ca: Image successfully pushed 
8254ff58b098: Image successfully pushed 
ec5f59360a64: Image successfully pushed 
2ce4ac388730: Image successfully pushed 
2eccda511755: Image successfully pushed 
5a14c1498ff4: Image successfully pushed 
8ffd698b4b9a: Image successfully pushed 
c9950e27e2bf: Image successfully pushed 
f5489e95a03b: Image successfully pushed 
13e9704168f6: Image successfully pushed 
d329e079a86b: Image successfully pushed 
9675842043c7: Image successfully pushed 
949a55b1c715: Image successfully pushed 
9205a67b7f7d: Image successfully pushed 
70bee8e8629f: Image successfully pushed 
78934e85029e: Image successfully pushed 
Pushing tag for rev [78934e85029e] on {http://107.191.40.91:5000/v1/repositories/haproxy0/tags/latest}

Still odd I got an incorrect username/password error..? Not a "could not find image" error for the un-tagged image?

FATA[0004] could not find image: no such id: 107.191.40.91:5000/shaped/ubuntu-base 
core@core0 ~ $ docker push shaped/haproxy
The push refers to a repository [shaped/haproxy] (len: 1)
Sending image list

Please login prior to push:
Username: 
FATA[0001] Error response from daemon: Registration: "Missing username field" 
core@core0 ~ $ 
core@core0 ~ $ docker push shaped/haproxy
The push refers to a repository [shaped/haproxy] (len: 1)
Sending image list

Please login prior to push:
Username: shaped
Password: 
Email: shaped
FATA[0002] Error response from daemon: Registration: "Wrong email format (it has to match \"[^@]+@[^@]+\\.[^@]+\")" 
core@core0 ~ $ docker push shaped/haproxy
The push refers to a repository [shaped/haproxy] (len: 1)
Sending image list

Please login prior to push:
Username: shaped
Password: 
Email: shaped@shaped.ca
FATA[0004] Error response from daemon: Wrong login/password, please try again

So in the end - it does work - however there's still the issue of Documentation, misleading outputs & error messages.

ForbiddenEra commented 9 years ago

I also should say I still can't drop the --insecure-registry even though I've added my CA to the host & to docker.. I showed how I generated above as well, any ideas?

ForbiddenEra commented 9 years ago

If I specify a port, docker uses https (whether that port is say, 5000 or 443)

If I don't docker defaults to http and port 80 - though it SEEMS to work if it finds SSL at 80 instead?

ForbiddenEra commented 9 years ago

Yeah, I can login but not push when ssl is on 80.. have to specify port manually. :(

ForbiddenEra commented 9 years ago

I may have not restarted my docker daemon after giving it the cert as the ssl is working properly now.

I hope, while not actually a real issue, this helps someone in the future setting up their registry.

danielschwartz commented 9 years ago

I'm actually running into this issue myself. I did a test push with no SSL and no basic auth enabled, everything works, so the registry itself works.

When I do --insecure-registry <host>:<port> I get this error:

x509: certificate signed by unknown authority. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry harrys.dyn-o-saur.com:8080` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at

while it's trying to post to /v1/_ping

When I do --insecure-registry https://<host>:<port> I get this error: x509: certificate signed by unknown authority while it's trying to post to /v1/users/

Meaning when I set https:// I get none of the extra messaging, suggesting that it's somehow set more correctly. However, in both formats, the commands don't actually work.

danielschwartz commented 9 years ago

I also just tried --insecure-registry="<host>:<port>" to an inverse of the above. Meaning I am now able to docker login <host>:<port> and have it give me the shorter error. Still though, cant move forward and login. Still getting the x509: certificate signed by unknown authority error.

dmp42 commented 9 years ago

@danielschwartz

your registry is working ok without authentication nor ssl - correct?
can you make it work ok with just SSL and NO authentication first? (so, no docker login)

danielschwartz commented 9 years ago

Yes
No, the issue is with the cert, not with the login. Because it is a self-signed cert it's failing, even though i've added the registry domain and port to --insecure-registry

dmp42 commented 9 years ago

@danielschwartz

why would you try to login if there is no auth for your registry?
save for docker login, can you docker push and pull successfully?
what version of docker are you using?
are you using the insecure registry argument for the docker daemon (not the cli)?

danielschwartz commented 9 years ago

Not using the docker login with no auth. When there is no auth for the registry it simply fails on the /_v1/ping call instead of the login call
Not with SSL enabled, I get the x509 error. Without SSL enabled, I can push/pull just fine
Docker 1.5.0 for the cli, the latest registry docker container, this was all pulled last night
Yup, using it on the daemon, not on the cli i've checked by looking at ps aux | grep docker and making sure the daemon is running with it

dmp42 commented 9 years ago

@danielschwartz please (with SSL enabled) curl -iv https://yourregistry:port/v1/_ping and:

check if curl is happy
copy the resulting output

ForbiddenEra commented 9 years ago

@danielschwartz My issue was not restarting the Docker daemon after providing the cert in /etc/docker/certs.d/...

Also, make sure that you're specifying the protocol & port.

I had issues not specifying the port specifically.

-------- Original message -------- From: Olivier Gambier notifications@github.com Date: 02-27-2015 2:36 PM (GMT-07:00) To: docker/docker-registry docker-registry@noreply.github.com Cc: Jai Boudreau jason@shaped.ca Subject: Re: [docker-registry] Following instructions in readme.md for test/dev/default registry not working (#945)

@danielschwartz please (with SSL enabled) curl -iv https://yourregistry:port/v1/_ping and:

check if curl is happy copy the resulting output — Reply to this email directly or view it on GitHub.

ForbiddenEra commented 9 years ago

Also,

EDIT: Apparently, I started typing something here..and I don't remember what. Sorry. Did you ever get it working @danielschwartz

mlhamel commented 9 years ago

I had this issue with docker and my certificate generated at StartSSL. I've fixed by following those steps:

http://www.startssl.com/?app=42, basically:

Fetch the Root CA and Class 1 Intermediate Server CA certificates:

$ wget http://www.startssl.com/certs/sub.class1.server.ca.pem

Create a unified certificate from your certificate and the CA certificates:

$ cat ssl.crt sub.class1.server.ca.pem > /etc/nginx/conf/ssl-unified.crt

And then use this new combined certificate in nginx !

docker-archive / docker-registry

Following instructions in readme.md for test/dev/default registry not working #945