search

sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.85k stars 2.14k forks source link

Docker Container Registry v2 API error 400 Bad Request Behind Nginx Reverse Proxy #1881

Closed SysLunix closed 4 years ago

SysLunix commented 5 years ago

Hello everyone, I come to post my problem after hours of searching without finding the cause.

The context:

Gitlab and the Docker Registry behind a Nginx reverse proxy on the host machine. The Gitlab seems to work well however the container registry returns me a 400 Bad Request error when I try a docker login.

My docker-composes.yml :

version: '2'

services:
  redis:
    restart: always
    image: sameersbn/redis:4.0.9-1
    command:
    - --loglevel warning
    volumes:
    - ./gitlab/redis:/var/lib/redis:Z

  postgresql:
    restart: always
    image: sameersbn/postgresql:10
    volumes:
    - ./gitlab/postgresql:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm

  gitlab:
    restart: always
    image: sameersbn/gitlab:11.8.2
    depends_on:
    - redis
    - postgresql
    ports:
    - "10080:80"
    - "10022:22"
    volumes:
    - ./gitlab:/home/git/data:Z
    - ./certs:/certs
    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Europe/Paris
    - GITLAB_TIMEZONE=Paris

    - GITLAB_HTTPS=true
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=gitlab.example.io
    - GITLAB_PORT=
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=
    - GITLAB_SECRETS_DB_KEY_BASE=secret
    - GITLAB_SECRETS_SECRET_KEY_BASE=secret
    - GITLAB_SECRETS_OTP_KEY_BASE=secret

    # Registry
    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST=registry.example.io
    - GITLAB_REGISTRY_PORT=443
    - GITLAB_REGISTRY_API_URL=http://registry:5000
    - GITLAB_REGISTRY_KEY_PATH=/certs/registry.key
    - GITLAB_REGISTRY_ISSUER=gitlab-issuer
    #- SSL_REGISTRY_KEY_PATH=/certs/registry.key
    #- SSL_REGISTRY_CERT_PATH=/certs/registry.crt

    - GITLAB_ROOT_PASSWORD=SECRET
    - GITLAB_ROOT_EMAIL=mail@example.io

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - GITLAB_EMAIL=gitlab@example.io
    - GITLAB_EMAIL_REPLY_TO=noreply@example.io
    - GITLAB_INCOMING_EMAIL_ADDRESS=reply@example.io

    # - GITLAB_BACKUP_SCHEDULE=daily
    # - GITLAB_BACKUP_TIME=01:00

    # - IMAP_ENABLED=false
    # - IMAP_HOST=imap.gmail.com
    # - IMAP_PORT=993
    # - IMAP_USER=mailer@example.com
    # - IMAP_PASS=password
    # - IMAP_SSL=true
    # - IMAP_STARTTLS=false

    # - OAUTH_ENABLED=false
    # - OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
    # - OAUTH_ALLOW_SSO=
    # - OAUTH_BLOCK_AUTO_CREATED_USERS=true
    # - OAUTH_AUTO_LINK_LDAP_USER=false
    # - OAUTH_AUTO_LINK_SAML_USER=false
    # - OAUTH_EXTERNAL_PROVIDERS=

    # - OAUTH_CAS3_LABEL=cas3
    # - OAUTH_CAS3_SERVER=
    # - OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
    # - OAUTH_CAS3_LOGIN_URL=/cas/login
    # - OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
    # - OAUTH_CAS3_LOGOUT_URL=/cas/logout

    # - OAUTH_GOOGLE_API_KEY=
    # - OAUTH_GOOGLE_APP_SECRET=
    # - OAUTH_GOOGLE_RESTRICT_DOMAIN=

    # - OAUTH_FACEBOOK_API_KEY=
    # - OAUTH_FACEBOOK_APP_SECRET=

    # - OAUTH_TWITTER_API_KEY=
    # - OAUTH_TWITTER_APP_SECRET=

    # - OAUTH_GITHUB_API_KEY=
    # - OAUTH_GITHUB_APP_SECRET=
    # - OAUTH_GITHUB_URL=
    # - OAUTH_GITHUB_VERIFY_SSL=

    # - OAUTH_GITLAB_API_KEY=
    # - OAUTH_GITLAB_APP_SECRET=

    # - OAUTH_BITBUCKET_API_KEY=
    # - OAUTH_BITBUCKET_APP_SECRET=

    # - OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
    # - OAUTH_SAML_IDP_CERT_FINGERPRINT=
    # - OAUTH_SAML_IDP_SSO_TARGET_URL=
    # - OAUTH_SAML_ISSUER=
    # - OAUTH_SAML_LABEL="Our SAML Provider"
    # - OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
    # - OAUTH_SAML_GROUPS_ATTRIBUTE=
    # - OAUTH_SAML_EXTERNAL_GROUPS=
    # - OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
    # - OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
    # - OAUTH_SAML_ATTRIBUTE_STATEMENTS_USERNAME=
    # - OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
    # - OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=

    # - OAUTH_CROWD_SERVER_URL=
    # - OAUTH_CROWD_APP_NAME=
    # - OAUTH_CROWD_APP_PASSWORD=

    # - OAUTH_AUTH0_CLIENT_ID=
    # - OAUTH_AUTH0_CLIENT_SECRET=
    # - OAUTH_AUTH0_DOMAIN=

    # - OAUTH_AZURE_API_KEY=
    # - OAUTH_AZURE_API_SECRET=
    # - OAUTH_AZURE_TENANT_ID=

  registry:
    image: registry:2.4.1
    restart: always
    expose:
    - "5000"
    ports:
    - "5000:5000"
    volumes:
    - ./gitlab/shared/registry:/registry
    - ./certs:/certs
    environment:
    - REGISTRY_LOG_LEVEL=info
    - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
    - REGISTRY_AUTH_TOKEN_REALM=https://gitlab.example.io/jwt/auth
    - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
    - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
    - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry.crt
    - REGISTRY_STORAGE_DELETE_ENABLED=true
    - REGISTRY_HTTP_SECRET=iamasecretregistry

registry key generated by : 

mkdir certs
cd certs
openssl req -new -newkey rsa:4096 > registry.csr
openssl rsa -in privkey.pem -out registry.key
openssl x509 -in registry.csr -out registry.crt -req -signkey registry.key -days 10000

reference : https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md

My Nginx Configuration :

For Gitlab : 

upstream gitlab {
  server 127.0.0.1:10080 fail_timeout=0;
}

server {
    listen 80;
    listen [::]:80 ipv6only=on;
    server_name gitlab.example.io;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ipv6only=on ssl http2;

    server_name gitlab.example.io;

    ssl_certificate /etc/letsencrypt/live/gitlab.example.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/gitlab.example.io/privkey.pem;

    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_timeout  5m;

    ## Individual nginx logs for this GitLab vhost
    access_log  /var/log/nginx/gitlab_access.log;
    error_log   /var/log/nginx/gitlab_error.log;

 location / {
        client_max_body_size 0;
        gzip off;

        ## https://github.com/gitlabhq/gitlabhq/issues/694
        ## Some requests take more than 30 seconds.
        proxy_read_timeout    300;
        proxy_connect_timeout 300;
        proxy_redirect        off;

        proxy_http_version    1.1;

        proxy_set_header Host                $http_host;
        proxy_set_header X-Real-IP           $remote_addr;
        proxy_set_header X-Forwarded-Ssl     on;
        proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto   $scheme;
        proxy_pass       http://gitlab;
    }
}

For the registry : 

server {
    root /dev/null;
    server_name registry.example.io;
    charset UTF-8;
    access_log /var/log/nginx/registry.example.io.access.log;
    error_log /var/log/nginx/registry.example.io.error.log;

    # Set up SSL only connections:
    listen *:443 ssl http2;
    ssl_certificate             /etc/letsencrypt/live/registry.example.io/fullchain.pem;
    ssl_certificate_key         /etc/letsencrypt/live/registry.example.io/privkey.pem;

    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_timeout  5m;

    client_max_body_size        0;
    chunked_transfer_encoding   on;

    location / {
        proxy_set_header  Host              $http_host;   # required for docker client's sake
        proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_read_timeout                  900;
        proxy_pass        http://localhost:5000;
    }
}

server {
    listen *:80;
    server_name  registry.example.io;
    server_tokens off; ## Don't show the nginx version number, a security best practice
    return 301 https://$http_host:$request_uri;
}

docker version / docker ps result : 

deploy@gitlab:~$ docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:34:04 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 05:59:55 2019
  OS/Arch:          linux/amd64
  Experimental:     false
deploy@gitlab:~$ docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                   NAMES
a41156f598c5        sameersbn/gitlab:11.8.2   "/sbin/entrypoint.sh…"   About an hour ago   Up About an hour    443/tcp, 0.0.0.0:10022->22/tcp, 0.0.0.0:10080->80/tcp   git_gitlab_1
a6c977843775        registry:2.4.1            "/bin/registry serve…"   About an hour ago   Up About an hour    0.0.0.0:5000->5000/tcp                                  git_registry_1
31d16352a458        sameersbn/postgresql:10   "/sbin/entrypoint.sh"    About an hour ago   Up About an hour    5432/tcp                                                git_postgresql_1
8d8147529c17        sameersbn/redis:4.0.9-1   "/sbin/entrypoint.sh…"   About an hour ago   Up About an hour    6379/tcp                                                git_redis_1

a result of curl -Lv https://registry.example.io/v2

curl -Lv https://registry.example.io/v2
*   Trying XXX.XXX.XXX.XXX...
* TCP_NODELAY set
* Connected to registry.example.io (XXX.XXX.XXX.XXX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=registry.example.io
*  start date: Mar 20 11:01:55 2019 GMT
*  expire date: Jun 18 11:01:55 2019 GMT
*  subjectAltName: host "registry.example.io" matched cert's "registry.example.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55a9aee44f20)
> GET /v2 HTTP/2
> Host: registry.example.io
> User-Agent: curl/7.64.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 301 
< server: nginx
< date: Thu, 21 Mar 2019 11:17:10 GMT
< content-type: text/html; charset=utf-8
< content-length: 39
< docker-distribution-api-version: registry/2.0
< location: /v2/
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< 
* Ignoring the response-body
* Connection #0 to host registry.example.io left intact
* Issue another request to this URL: 'https://registry.example.io/v2/'
* Found bundle for host registry.example.io: 0x55a9aee44170 [can multiplex]
* Re-using existing connection! (#0) with host registry.example.io
* Connected to registry.example.io (137.74.49.142) port 443 (#0)
* Using Stream ID: 3 (easy handle 0x55a9aee44f20)
> GET /v2/ HTTP/2
> Host: registry.example.io
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/2 401 
< server: nginx
< date: Thu, 21 Mar 2019 11:17:10 GMT
< content-type: application/json; charset=utf-8
< content-length: 87
< docker-distribution-api-version: registry/2.0
< www-authenticate: Bearer realm="https://gitlab.example.io/jwt/auth",service="container_registry"
< x-content-type-options: nosniff
< 
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host registry.example.io left intact

docker login result : 

docker login registry.example.io
Username: root
Password: 
Error response from daemon: login attempt to https://registry.example.io/v2/ failed with status: 400 Bad Request

I tried with the Gitlab account and also with a Personal Token Access and a Deploy Key, same error...

If you have an idea and need more information I thank you in advance

r900 commented 5 years ago

i'm not 100% sure if this is the correct way to do this, but here's how i got it working:

1) exposed port 5000 of the gitlab container in the docker-compose.yml. the registry has no port exposed at all, everything goes through gitlab.

2) the gitlab service has

    external_links:
    - "registry:registry.example.com"

the registry service

    external_links:
    - "gitlab:gitlab.example.com"

not sure if this is still necessary, but it helped at some point in the past ^^

3) the upstream of the registry:443 nginx config has a different upstream: https://registry.example.com:5000 instead of http://localhost:5000

4) i'm using the proper certs for registry.example.com in the certs folder, not a self-signed one

5) in the docker-compose.yml:

hth

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.

MohdRashid01 commented 4 years ago

I cannot delete private or public repository from docker hub

i try with this command curl -s -v -H “Authorization: JWT ${HUB_TOKEN}” -X DELETE “https://hub.docker.com/v2/repositories/mohd/test2/”

got error 400 BAD REQUEST Connection #0 to host hub.docker.com left intact {“detail”: “There was an error. The repository was not deleted.”}