Docker Registry 2.0.1 and NGI Proxy with LDAP Authentication Issue on GET /v1/users

deviscalio commented 8 years ago

Hello,

I'm facing some problems to setup a docker registry with an active directory. I've setup with success the authentication with active directory, but after login to registry this fails with these messages:

On docker side:

registry_1 | time="2015-08-05T14:08:52.073707203Z" level=info msg="endpoint local-8082 disabled, skipping" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1 registry_1 | time="2015-08-05T14:08:52.073774138Z" level=info msg="endpoint local-8083 disabled, skipping" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1 registry_1 | time="2015-08-05T14:08:52.073821358Z" level=info msg="using inmemory layerinfo cache" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1 registry_1 | time="2015-08-05T14:08:52.073843722Z" level=info msg="listening on :5000" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1 registry_1 | time="2015-08-05T14:08:52.073924198Z" level=info msg="debug server listening localhost:5001" nginx_1 | nginx: [emerg] http_auth_ldap: parse_require in /etc/nginx/nginx.conf:22 nginx_1 | nginx: [emerg] http_auth_ldap: Setting satisfy all in /etc/nginx/nginx.conf:23 registry_1 | time="2015-08-05T14:09:06.38493673Z" level=info msg="response completed" environment=development http.request.host=localhost http.request.id=88e7ecdf-1306-4b7c-be5c-c2d8aa64c0ac http.request.method=GET http.request.remoteaddr=172.17.42.1 http.request.uri="/v1/_ping" http.request.useragent="docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="115.795µs" http.response.status=404 http.response.written=19 instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1 registry_1 | 172.17.0.12 - - [05/Aug/2015:14:09:06 +0000] "GET /v1/_ping HTTP/1.0" 404 19 "" "docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64" registry_1 | time="2015-08-05T14:09:06.402938569Z" level=info msg="response completed" environment=development http.request.host=localhost http.request.id=a7ae38ac-bb89-43e5-ac2e-588e6594254e http.request.method=GET http.request.remoteaddr=172.17.42.1 http.request.uri="/v1/users/" http.request.useragent="docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="112.529µs" http.response.status=404 http.response.written=19 instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1 registry_1 | 172.17.0.12 - - [05/Aug/2015:14:09:06 +0000] "GET /v1/users/ HTTP/1.0" 404 19 "" "docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64"

On User side after a docker login: Username: test Password: Email: test@test.it Error response from daemon: Login: 404 page not found (Code: 404; Headers: map[Server:[nginx/1.7.12] Date:[Wed, 05 Aug 2015 14:09:06 GMT] Content-Type:[text/plain; charset=utf-8] Content-Length:[19] Connection:[keep-alive] Docker-Distribution-Api-Version:[registry/2.0 registry/2.0.1]])

Then I cannot access to registry.

My nginx.conf is:

worker_processes 1;

events { worker_connections 1024; }

http {

upstream docker-registry {
    server docker-registry:5000;
}

ldap_server ldapserver {
url ldap://**********:389/OU=Users,OU=*******,DC=*******,DC=it?samaccountname?sub?(objectClass=user);
    #binddn "cn=admin,dc=example,dc=com";
    binddn ************;
    binddn_passwd **********
    group_attribute uniquemember;
    group_attribute_is_dn on;
    #require group 'cn=docker,ou=groups,dc=example,dc=com';
    require valid_user;
    satisfy all;
}

server {

    listen 443;
    server_name *************;

    error_log /var/log/nginx/error.log debug;
    access_log /var/log/nginx/access.log;

    ssl on;
    ssl_certificate /etc/ssl/docker/docker-registry.crt;
    ssl_certificate_key /etc/ssl/docker/docker-registry.key;

    client_max_body_size 0;

    chunked_transfer_encoding on;

add_header Docker-Distribution-Api-Version registry/2.0.1 always;

    location / {
        auth_ldap "Forbidden";
        auth_ldap_servers ldapserver;
        include docker-registry.conf;
    }

    location ~* /v1/repositories/(?<namespace>([^/]*))/ {
        auth_ldap "Forbidden";
        auth_ldap_servers ldapserver; 
        set $deny_write_request "";
        if ($request_method = PUT) {
            set $deny_write_request "W";
        }           
        if ($request_method = DELETE) { 
            set $deny_write_request "W";
        }           
        if ($remote_user != $namespace) {
            set $deny_write_request "${deny_write_request}A";
        }           
        if ($deny_write_request = 'WA') {
            return 401;     
        }           
        include docker-registry.conf;
    }       

location /v2/ {
        # Do not allow connections from docker 1.5 and earlier
        # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
        if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
                return 404;
        }

            auth_ldap "Forbidden";
            auth_ldap_servers ldapserver; 
    #auth_basic off;
            include docker-registry.conf;
}

    location /_ping {
        auth_basic off;
        include docker-registry.conf;
    }

    location /v1/_ping {
        auth_basic off;
        include docker-registry.conf;
    }

    location /v1/search {
        auth_basic off;
        include docker-registry.conf;
    }

}

}

docker-registry.conf is

   proxy_pass                       http://docker-registry;
   proxy_set_header  Docker-Distribution-Api-Version registry/2.0.1;
   proxy_set_header  Host           $http_host;   # required for docker client's sake
   proxy_set_header  X-Real-IP      $remote_addr; # pass on real client's IP
   proxy_set_header  Authorization  ""; # see https://github.com/dotcloud/docker-registry/issues/170
   proxy_read_timeout               900;

Any suggestions?

dmp42 commented 8 years ago

Hi,

I strongly recommend you move away from registry:1 - or at least first try to make things work for registry:2 only.

Try to follow the instructions in the official documentation, specifically: https://github.com/docker/distribution/blob/master/docs/authentication.md and see if you can setup basic auth successfully.

Then when you are done, add ldap.

deviscalio commented 8 years ago

Hello,

How I can move away from registry:1 ? I'm already using image registry:2.0.1. Here my docker-compose file:

################################ nginx: image: h3nrik/registry-ldap-auth ports:

"443:443" links:
registry:docker-registry volumes:
$PWD:/etc/ssl/docker:ro
$PWD:/etc/nginx:ro restart: always

registry: image: registry:2.0.1 hostname: "docker-registry" volumes:

/docker_images:/tmp/registry-dev ports:
"5000" restart: always #############################

Moreover your link in your response is not valid.

dmp42 commented 8 years ago

Hi @deviscalio

The documentation page was moved.

Have a look here: https://github.com/docker/distribution/blob/master/docs/nginx.md

And if you don't want to use v1, please remove the v1 sections from your configuration (or start from the exemple in the documentation).

deviscalio commented 8 years ago

Many thanks Oliver.

When you say "... please remove the v1 sections from your configuration", do you mean remove from nginx configuration or from registry configuration? In the second case which is this file?

dmp42 commented 8 years ago

@deviscalio I suggest you start from scratch, using the documentation I linked. Once you get it working, you move ahead and add your LDAP snippets.

deviscalio commented 8 years ago

Hello Olivier,

I've followed the guide and I've had a successful installation of registry. Then, as you suggested, I've added LDAP configuration, but of course the nginx image is wrong because I've an error : "unknown directive "ldap_server".

I've tried with h3nrik/registry-ldap-auth image but now I've this error:

Error response from daemon: no successful auth challenge for https://localhost:5043/v2/ - errors: [basic auth attempt to https://localhost:5043/v2/ realm "Forbidden" failed with status: 401 Unauthorized]

I've substituted in registry.conf: auth_basic "Registry realm"; auth_basic_user_file /etc/nginx/conf.d/htpasswd; with auth_ldap "Forbidden"; auth_ldap_servers ldapserver;

deviscalio commented 8 years ago

Hello Oliver,

I've resolved! I put here the ngix.conf used, hoping it helps someone:

########################################### worker_processes 1;

events { worker_connections 1024; }

http {

upstream docker-registry {
    server docker-registry:5000;
}

ldap_server ldapserver {
url ldap://xxxxxxxxx:389/OU=Users,DC=xxxxx,DC=it?samaccountname?sub?(objectClass=user);
    binddn xxxxx@xxxxxxxxxx.it;
    binddn_passwd xxxxxxxxxxxx
    group_attribute uniquemember;
    group_attribute_is_dn on;
}

server {

    listen 443;
    server_name xxxxxxxxxxxxxxxxxx;

    error_log /var/log/nginx/error.log debug;
    access_log /var/log/nginx/access.log;

    ssl on;
    ssl_certificate /etc/ssl/docker/docker-registry.crt;
    ssl_certificate_key /etc/ssl/docker/docker-registry.key;

    client_max_body_size 0;

    chunked_transfer_encoding on;

location /v2/ {
        # Do not allow connections from docker 1.5 and earlier
        # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
        if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
                return 404;
        }

            auth_ldap "Forbidden";
            auth_ldap_servers ldapserver; 
            add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

    proxy_pass                          http://docker-registry;
        proxy_set_header  Host              $http_host;   # required for docker client's sake
        proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_read_timeout                  900;
}
}

} ###########################################

And docker-compose.yml used:

########################################### nginx: image: h3nrik/registry-ldap-auth ports:

"443:443" links:
registry:docker-registry volumes:
$PWD/certs:/etc/ssl/docker:ro
$PWD/conf:/etc/nginx:ro restart: always

registry: image: registry:2.0 hostname: "docker-registry" volumes:

/docker_images:/tmp/registry-dev ports:
"5000" restart: always ###########################################

nagarjung commented 8 years ago

@deviscalio : Could you please help me with the link you followed. I am trying to setup private registry authentication with SSL self signed certificates + LDAP(not a container, existing server ).

Basic authentication using SSL is working fine. But unable to figure out configurations with LDAP.

phucvinh52 commented 8 years ago

Hi @deviscalio i just have got this error. Could you tell me how to resolve this problem? Thank you.

deviscalio commented 8 years ago

https://github.com/tierratelematics/existing-ldap-docker-registry

Bart187 commented 8 years ago

Hi @deviscalio

I'm trying to use Your image, but I got that error:

[root@D-L-TOOLS docker]# docker-compose up -d
Recreating b06951df6d83_b06951df6d83_b06951df6d83_docker_registry_1

ERROR: for registry  No such image: sha256:34bccec547938a300eb4a3dc8161ff71ec40ccddc8341961ddcf57fcbf8a73f5
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 63, in main
AttributeError: 'ProjectError' object has no attribute 'msg'
docker-compose returned -1

Somebody erased some of images? How can I obey it? If I check docker images I have h3nrik/registry-ldap-auth and registry 2.3.1 downloaded. The only thing which I have changed in Your files were the paths that You mentioned here:

https://github.com/tierratelematics/existing-ldap-docker-registry

###############################################################################

My docker-compose.yml file looks like this:

nginx:
  image: h3nrik/registry-ldap-auth
  ports:
    - "443:443"
  links:
    - registry:docker-registry
  volumes:
    - $PWD/certs:/etc/ssl/docker:ro
    - $PWD/conf:/etc/nginx:ro
  restart: always

registry:
  image: registry:2.3.1
  hostname: "docker-registry"
  environment:
    - REGISTRY_DELETE_ENABLED=true
  volumes:
    - /data/docker_registry/docker:/var/lib/registry
  ports:
    - "5000"
  restart: always

###############################################################################

Nginx conf looks like:

worker_processes  1;

events {
    worker_connections  1024;
}

http {

    upstream docker-registry {
        server docker-registry:5000;
    }

    ldap_server ldapserver {
        url "ldaps://xxxxxxxxxxxxxxx:636/OU=xxxx,DC=xxxx,DC=local?sAMAccountName?sub?(&(memberOf:1.2.840.113556.1.4.1941:=cn=xxxxxx,OU=User_Groups,OU=Groups,OU=GLOB000,OU=Global,OU=ROOT,DC=xxxx,DC=local)(objectClass=person))" SSL;
        binddn "CN=xxxxxxxxxxx,OU=Service_Accounts,OU=Admins,OU=GLOB000,OU=Global,OU=xxxx,DC=xxxxxx,DC=local";
        binddn_passwd xxxxxxxxxxxxxx
        group_attribute uniquemember;
        group_attribute_is_dn on;
    }

    server {

        listen 443;
        server_name <my-server-name>;

        error_log /var/log/nginx/error.log debug;
        access_log /var/log/nginx/access.log;

        ssl on;
        ssl_certificate /etc/ssl/docker/docker-registry.crt;
        ssl_certificate_key /etc/ssl/docker/docker-registry.key;

        client_max_body_size 0;

        chunked_transfer_encoding on;

        location / {
                return 301 https://<my-server-name>/v2;
        }

        location /v2/ {
                # Do not allow connections from docker 1.5 and earlier
                # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
                if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
                        return 404;
                }

                auth_ldap "Forbidden";
                auth_ldap_servers ldapserver;
                add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

                proxy_pass                          http://docker-registry;
                proxy_set_header  Host              $http_host;   # required for docker client's sake
                proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
                proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
                proxy_set_header  X-Forwarded-Proto $scheme;
                proxy_read_timeout                  900;
        }
    }

}

###############################################################################

I have also overwritten Your certs in pwd/certs catalog with my self signed certs which were previously worked with my other private registry (without authentication). Your compose should work, but some old container could mess things up or missing? I have removed with docker rmi -f all unused images of registries and nginx servers, so docker is clean now...

Please help.

Bartosz

Bart187 commented 8 years ago

I found the problem (I should remove all the containers that were in memory), but another one came out. During the last part - when I try to log in to server I have:

docker login https://"my_address".local
Username: "my_user"
Password:
Error response from daemon: Missing client certificate domain.cert for key domain.key

Seems like my self-signed certs are not visible for environment. But the paths are ok:

From Nginx conf:

    ssl_certificate /etc/ssl/docker/docker-registry.crt;
    ssl_certificate_key /etc/ssl/docker/docker-registry.key;

From docker-compose.yml:

  volumes:
    - /data/docker_registry/docker/certs:/etc/ssl/docker:ro
    - /data/docker_registry/docker/conf:/etc/nginx:ro

[root@D-L-TOOLS certs]# pwd
/data/docker_registry/docker/certs
[root@D-L-TOOLS certs]# ll
total 8
-rwx------. 1 root root 2187 Jun  1 09:56 docker-registry.crt
-rwx------. 1 root root 3272 Jun  1 09:56 docker-registry.key

Still there is something wrong:/

deviscalio commented 8 years ago

Try to connect to nginx container:

docker exec -it _containerid /bin/bash

and check if files are present.

Bart187 commented 8 years ago

Hi,

Here is an output from nginx container certificates path:

root@3e2e82d415bd:/etc/ssl/docker# ls -la
total 12
drwx------. 2 root root 4096 Jun  1 06:51 .
drwxr-xr-x. 5 root root   63 Jun  1 12:50 ..
-rwx------. 1 root root 2187 Jun  1 06:56 docker-registry.crt
-rwx------. 1 root root 3272 Jun  1 06:56 docker-registry.key
root@3e2e82d415bd:/etc/ssl/docker# pwd
/etc/ssl/docker

Checked the docker-registry.crt file and it seems ok. The cert is the same as in the /data/docker_registry/docker/certs on my server.

I saw also got that error:

Error response from daemon: Missing client certificate domain.cert for key domain.key when I login with correct and wrong credentials...

Bart187 commented 8 years ago

Got the same when I'm trying to push image to my registry:

 docker push d-l-xxxxx.xxxxx.local:5000/ubuntu
 The push refers to a repository [d-l-xxxxxx.xxxxx.local:5000/ubuntu]
 Get https://d-l-xxxxx.xxxxxx.local:5000/v1/_ping: dial tcp 10.xxx.xx.xx:5000: getsockopt: connection refused

 docker push d-l-xxxxx.xxxxxx.local/ubuntu
 Error response from daemon: Missing client certificate domain.cert for key domain.key

But after some debugging I think I know what could be wrong:

      nginx_1     | 2016/06/01 12:50:21 [emerg] 8#0: http_auth_ldap: host not found in LDAP hostname  "ldap.xxxxxx.com" in /etc/nginx/nginx.conf:15
      nginx_1     | 2016/06/01 13:02:08 [debug] 13#0: epoll add event: fd:8 op:1 ev:00002001
      nginx_1     | 2016/06/01 13:07:08 [error] 13#0: http_auth_ldap: ldap_result() failed (-1: Can't contact LDAP server)
      nginx_1     | 2016/06/02 06:09:48 [error] 13#0: shutdown() failed (107: Transport endpoint is not connected)
      nginx_1     | 2016/06/02 06:09:52 [alert] 8#0: worker process 13 exited on signal 9
      nginx_1     | 2016/06/02 06:14:00 [debug] 13#0: epoll add event: fd:8 op:1 ev:00002001

I will check the it from LDAP side. But it seems that something should be enabled to use TLS... Any thoughts?

I saw that on other servers (with apache and ldap configuration) there is something like that:

    <IfModule ldap_module>
      LDAPTrustedMode TLS
      LDAPVerifyServerCert Off
    </IfModule>

Is Nginx using TLS too (is it possible)? Can I configure it?

Bart187 commented 8 years ago

Hi Again,

I have added to configuration two lines (where ldap.pem is CAcertificate from LDAP server):

ssl_check_cert off;
ssl_ca_file /etc/ssl/docker/ldap.pem;

And now I have something like this:

docker-compose up
Starting docker_registry_1
Starting docker_nginx_1
Attaching to docker_registry_1, docker_nginx_1
registry_1  | time="2016-06-02T09:21:26Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_DELETE_ENABLED"
registry_1  | time="2016-06-02T09:21:26Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="redis not configured" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="listening on [::]:5000" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="Starting upload purge in 18m0s" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
nginx_1     | 2016/06/02 09:18:49 [debug] 13#0: epoll add event: fd:8 op:1 ev:00002001
nginx_1     | 2016/06/02 09:21:27 [debug] 13#0: epoll add event: fd:8 op:1 ev:00002001

But got the same issue: Error response from daemon: Missing client certificate domain.cert for key domain.key

When I use it as that:

ssl_check_cert on;
ssl_ca_file /etc/ssl/docker/ldap.pem;

I have:

nginx_1     | nginx: [emerg] http_auth_ldap: 'ssl_cert_check': cannot verify remote certificate's domain name because your version of OpenSSL is too old. Please install OpenSSL >= 1.02 and recompile nginx. in /etc/nginx/nginx.conf:20
nginx_1     | 2016/06/02 09:25:23 [emerg] 10#0: http_auth_ldap: 'ssl_cert_check': cannot verify remote certificate's domain name because your version of OpenSSL is too old. Please install OpenSSL >= 1.02 and recompile nginx. in /etc/nginx/nginx.conf:20
nginx_1     | 2016/06/02 09:25:23 [debug] 13#0: epoll add event: fd:9 op:1 ev:00002001

But I have downloaded and installed new openssl version:

openssl version
OpenSSL 1.0.2g  1 Mar 2016

Do I need to tell nginx to look for openssl? It's configured to be in a default path: which openssl /bin/openssl

I think I'm near the end, as I don't have ldap errors any more...

Bart187 commented 8 years ago

I wanted to workaround that issue with openssl and missing certs by using:

 ldap_server ldapserver {
         url ldap://xxxxxx.xxxxx.local:389/OU=xxxx,DC=xxxxx,DC=local?sAMAccountName?sub?(&(memberOf:1.2.840.113556.1.4.1941:=cn=xxxxxxx,OU=xxxxxx,OU=xxxxxx,OU=xxxxx,OU=xxxxx,OU=xxxxx,DC=xxxxxx,DC=local)(objectClass=person));
        binddn "CN=xxxxxxxx,OU=xxxxxxxx,OU=Admins,OU=xxxxxxx,OU=xxxxxx,OU=xxxxx,DC=xxxxxx,DC=local";
        binddn_passwd xxxxxxxxxxx
        group_attribute uniquemember;
        group_attribute_is_dn on;
        #ssl_check_cert on;
        #ssl_ca_file /etc/ssl/docker/ldap.pem;
        #ssl_ca_dir /etc/ssl/docker/;

I have tested ldap connection with ldapserarch - it's running. Simple docker registry (without unsecure registry option) is working with that self-signed cert (also from other servers)...

But at the end I have been asked for certificates, no matter if I'm login with good or bad credentials:

docker login https://<my_domain_address>
Username: xxxxxx
Password:
Error response from daemon: Missing client certificate domain.cert for key domain.key

Bart187 commented 8 years ago

After upgrade openSSL and then downgrade to original one the error has vanished. It also seems like docker needs to have address with port on the end, because https:// is not enough. I've managed to login and push images after typing:

docker login https://<my_domain_address>:443

Thank You very much for Your help. My docker works! How can I make docker ask for login everytime somebody wants to push image? I have to logout everytime?

docker-archive / docker-registry

Docker Registry 2.0.1 and NGI Proxy with LDAP Authentication Issue on GET /v1/users #1026