Gitlab Registry port and host clarification

xxorax commented 7 years ago

I spend a lot of time to understand why I cannot contact to the Gitlab Registry (defined by $GITLAB_REGISTRY_HOST:$GITLAB_REGISTRY_PORT). I did not defined SSL_REGISTRY_CERT_PATH and SSL_REGISTRY_KEY_PATH, because I'm running behind an external proxy which handle HTTPS. So in this case, you only have to proxy to the Docker Registry (port 5000, from the registry:2 docker image), and so the GITLAB_REGISTRY_HOST and GITLAB_REGISTRY_PORT is only here to inform the gitlab users what is the Docker Registry domain..

I thing this should be mentioned somewhere, like "You can connect to the Docker Registry directly if you handle SSL yourself, and so these param only define the Docker Registry endpoint in the registry tab of a Gitlab project. When you will try to connect to the Docker Registry, it will tell your docker to connect to the Gitlab JWT auth service to get authenticated".

Actualy, even the Gitlab offical doc is very confuse : does Gitlab have it own Registry-app which is proxy the Docker Registry ? No : it's just a simple proxy defined in nginx. So there is no difference between docker login registry.gitlab.example.com and docker login registry:5000 except that you expect to have SSL enabled on registry.gitlab.example.com (because proxied by nginx with ssl).

In fact there is no difference with the schema of the offical doc except this nginx proxy to add SSL over the registry. And so you could even set the domain name of the registry as the same as the gitlab domain (gitlab.example.com) if you handle correctly the request in your proxy (for example by detecting ^/v2 in the request, or/and the docker user-agent string).

Pegasuz commented 7 years ago

So if I use nginx-proxy with the letsencrypt companion, I cannot get the registry to work?

xxorax commented 7 years ago

What nginx-proxy do you use ? If you use the config file provided, then you have change it to include letsencrypt, or you can put an other proxy over it. But the best way is to got your own nginx-proxy (with https) and link it directly to the Docker Registry (port 5000, no-https).

Pegasuz commented 7 years ago

I'm using https://github.com/jwilder/nginx-proxy and https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion

I got it to work now. For future reference I will leave my config here:

gitlab:
  // A lot of content skipped 
  environment:
    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST=registry.myhostname.be
    - GITLAB_REGISTRY_PORT=
    - GITLAB_REGISTRY_API_URL=http://registry:5000
    - GITLAB_REGISTRY_KEY_PATH=/home/git/data/registry.key

registry:
    restart: always
    image: registry:2.4.1
    volumes:
    - ./gitlab/shared/registry:/registry
    - ./certs:/certs
    environment:
    - VIRTUAL_HOST=registry.myhostname.be
    - VIRTUAL_PORT=5000
    - LETSENCRYPT_EMAIL=my@email.com
    - LETSENCRYPT_HOST=registry.myhostname.be
    - REGISTRY_LOG_LEVEL=info
    - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
    - REGISTRY_AUTH_TOKEN_REALM=https://gitlab.myhostname.be/jwt/auth
    - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
    - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
    - REGISTRY_STORAGE_DELETE_ENABLED=true
    - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry-auth.crt

Routhinator commented 7 years ago

A confusing solution @Pegasuz - The Gitlab container registry is internal to Gitlab. In your solution you are enabling the internal registry AND creating a defacto registry using the official docker registry container. Which one is actually answering? Looks like it's the docker registry and not the Gitlab registry.

xxorax commented 7 years ago

This is what it's confusing : there is NO gitlab registry, it's only an authentification service. As I explain in my comment, look at the schema on this page : https://docs.docker.com/registry/spec/auth/token/

solidnerd commented 7 years ago

Yep. This is true.

solidnerd commented 7 years ago

But GitLab calls it GitLab Container Registry.

Pegasuz commented 7 years ago

@xxorax is correct. This is why an external docker registry is needed. Hence my setup.

niksubramanian commented 7 years ago

@Pegasuz any chance of you sharing your full docker-compose setup (minus anything sensitive, obviously)? I'm having some trouble with the same setup as yours (nginx-proxy, letsencrypt, registry & gitlab).

Thanks in advance!

Pegasuz commented 7 years ago

@niksubramanian Here you go:


postgresql:
  restart: always
  image: sameersbn/postgresql:9.6-2
  environment:
    - DB_USER=username
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm
  volumes:
    - ./postgresql:/var/lib/postgresql
gitlab:
  restart: always
  image: sameersbn/gitlab:latest
  links:
    - redis:redis
    - postgresql:postgresql
    - registry:registry
  ports:
    - "9022:22"
  environment:
    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=username
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - DEBUG=false
    - TZ=Europe/Brussels
    - GITLAB_TIMEZONE=Brussels
    - VIRTUAL_HOST=gitlab.domain.be
    - LETSENCRYPT_EMAIL=user@domain.be
    - LETSENCRYPT_HOST=gitlab.domain.be

    - GITLAB_HTTPS=true
    - SSL_SELF_SIGNED=false
    - GITLAB_SECRETS_DB_KEY_BASE=long-and-random-fdgfdsgsdfgsdfgds
    - GITLAB_SECRETS_SECRET_KEY_BASE=long-and-random-alphanumeric-string
    - GITLAB_SECRETS_OTP_KEY_BASE=long-and-random-alphanumeric-string

    - GITLAB_HOST=gitlab.domain.be
    - GITLAB_PORT=443
    - GITLAB_SSH_PORT=9022
    - GITLAB_RELATIVE_URL_ROOT=

    - GITLAB_ROOT_PASSWORD=rootpassword
    - GITLAB_ROOT_EMAIL=user@domain.be

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - GITLAB_EMAIL=gitlab@domain.be
    - GITLAB_EMAIL_REPLY_TO=gitlab@domain.be
    - GITLAB_INCOMING_EMAIL_ADDRESS=gitlab@domain.be

    - GITLAB_BACKUP_SCHEDULE=disable
    - GITLAB_BACKUP_TIME=01:00

    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST=registry.domain.be
    - GITLAB_REGISTRY_PORT=
    - GITLAB_REGISTRY_API_URL=http://registry:5000
    - GITLAB_REGISTRY_KEY_PATH=/home/git/data/registry.key
  volumes:
    - ./gitlab:/home/git/data
    - ./logs:/var/log/gitlab
    - ./certs:/certs
redis:
  restart: always
  image: sameersbn/redis:latest
  volumes:
    - ./redis:/var/lib/redis

runner:
  image: sameersbn/gitlab-ci-multi-runner:latest
  volumes:
    - ./runner-gitlab:/home/gitlab_ci_multi_runner/data
    - /var/run/docker.sock:/var/run/docker.sock
    - ~/.docker:/root/.docker
    - /home/user/.ssh:/home/user/.ssh
  environment:
    - CI_SERVER_URL=https://gitlab.domain.be/ci
    - RUNNER_TOKEN=runnertoken
    - RUNNER_DESCRIPTION=DockerRunner
    - RUNNER_EXECUTOR=docker
  restart: always
  privileged: true

registry:
    restart: always
    image: registry:2.4.1
    volumes:
    - ./gitlab/shared/registry:/registry
    - ./certs:/certs
    environment:
    - VIRTUAL_HOST=registry.domain.be
    - VIRTUAL_PORT=5000
    - LETSENCRYPT_EMAIL=user@domain.be
    - LETSENCRYPT_HOST=registry.domain.be
    - REGISTRY_LOG_LEVEL=info
    - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
    - REGISTRY_AUTH_TOKEN_REALM=https://gitlab.domain.be/jwt/auth
    - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
    - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
    - REGISTRY_STORAGE_DELETE_ENABLED=true
    - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry-auth.crt
`

thiesschneider commented 7 years ago

hey @Pegasuz,

how do you get the cert for the registry in the right folder? Do you map your letsencrypt container in the gitlab cert folder?

thx in advance

mikehaertl commented 7 years ago

And so you could even set the domain name of the registry as the same as the gitlab domain (gitlab.example.com) if you handle correctly the request in your proxy (for example by detecting ^/v2 in the request, or/and the docker user-agent string).

@xxorax I also use my custom nginx proxy. Could you help me clarify two things?

You say if the same domain is used it should detect ^/v2 or something. But can't this be avoided by using a different public end port for the registry (using a dedicated server{ } block in nginx)?
I'm still confused about REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE and GITLAB_REGISTRY_KEY_PATH. What domain name is this certificate for? And why is this needed at all if we use our own Nginx?

mikehaertl commented 7 years ago

Ok, I've managed it.

@solidnerd I think, the manual is missing a clear example for how to set up the registry if you already have a Nginx as reverse proxy. How about adding the following to the documenation:

Setting up the Registry with Your Own Secured Nginx

If you already have a Nginx server as reverse proxy and manage your own certififcates there (e.g. with Letsencrypt) the setup can be simplified.

We assume the following URLs for your Gitlab installation and your registry respectively:

git.example.com
registry.example.com

1. Create auth tokens

The details entered during the creation process (domain name, etc.) are irrelevant:

mkdir certs
cd certs
openssl req -new -newkey rsa:4096 > registry.csr
openssl rsa -in privkey.pem -out registry.key
openssl x509 -in registry.csr -out registry.crt -req -signkey registry.key -days 10000

2. Update docker-compose.yml

First add the configuration for the registry container to your docker-compose.yml.

    registry:
        image: registry:2.6.2
        restart: unless-stopped
        expose:
            - "5000"
        ports:
            - "5000:5000"
        volumes:
            - ./gitlab/shared/registry:/registry
            - ./certs:/certs
        environment:
            - REGISTRY_LOG_LEVEL=info
            - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
            - REGISTRY_AUTH_TOKEN_REALM=https://git.example.de/jwt/auth
            - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
            - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
            - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry.crt
            - REGISTRY_STORAGE_DELETE_ENABLED=true

Then update the volumes and environment sections of your gitlab container:

    gitlab:
        environment:
            # ...
            # Registry
            - GITLAB_REGISTRY_ENABLED=true
            - GITLAB_REGISTRY_HOST=registry.example.de
            - GITLAB_REGISTRY_PORT=443
            - GITLAB_REGISTRY_API_URL=http://registry:5000
            - GITLAB_REGISTRY_KEY_PATH=/certs/registry.key

        volumes:
            - ./gitlab:/home/git/data
            - ./certs:/certs

3. Nginx site configuration

server {
    root /dev/null;
    server_name registry.example.de;
    charset UTF-8;
    access_log /var/log/nginx/registry.example.de.access.log;
    error_log /var/log/nginx/registry.example.de.error.log;

    # Set up SSL only connections:
    listen *:443 ssl http2;
    ssl_certificate             /etc/letsencrypt/live/registry.example.de/fullchain.pem;
    ssl_certificate_key         /etc/letsencrypt/live/registry.example.de/privkey.pem;

    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_session_timeout  5m;

    client_max_body_size        0;
    chunked_transfer_encoding   on;

    location / {
        proxy_set_header  Host              $http_host;   # required for docker client's sake
        proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_read_timeout                  900;
        proxy_pass          http://localhost:5000;
    }
}

server {
    listen *:80;
    server_name  registry.example.com;
    server_tokens off; ## Don't show the nginx version number, a security best practice
    return 301 https://$http_host:$request_uri;
}

solidnerd commented 7 years ago

@mikehaertl Feel free to make a PR for your last entry. I think would be nice for everyone else who need this. After this I think we can close this issue.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.

sameersbn / docker-gitlab