search

sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.87k stars 2.14k forks source link

Gitlab registry push fails #1246

Closed ChessSpider closed 7 years ago

ChessSpider commented 7 years ago

Hi,

I have a really weird problem I just can't figure out. I can login onto the docker registry but the push fails. I use gitlab for token-authentication with a self-signed certificate. Both gitlab and the registry are running behind a haproxy which does SSL off-loading. I placed my docker-compose.yml at the end.

I basically tried all things I could think of, but I really dont see anything wrong in the logs and many empty files are created in the registry folder.

Can you help me fix this?

IP 10.123.12.1 is the frontend haproxy IP.

machiel@machiel-VirtualBox:~/company_name/docker/postfix$ docker login -u machiel.surname@company_name.com -p password_for_demonstration_purposes registry.gitlab.example.com:5000
Login Succeeded
machiel@machiel-VirtualBox:~/company_name/docker/postfix$ docker push registry.gitlab.example.com:5000/company_name/postfix/image
The push refers to a repository [registry.gitlab.example.com:5000/company_name/postfix/image]
fa39aa088576: Retrying in 1 second 
bbb283a6cb0a: Retrying in 1 second 
9f28e853cb78: Retrying in 1 second 
3cd392719baf: Retrying in 1 second 
7fb34924ce7f: Retrying in 1 second 
3e5c31a7c94c: Waiting 
d8b353eb3025: Waiting 
f2e85bc0b7b1: Waiting 
fc9e1e5e38f7: Waiting 
fe9a3f9c4559: Waiting 
6a8bf8c8edbd: Waiting 
^C

At some point it just fails, printing EOF.

At the PUSH I do see a proper authentication in gitlab's log/production.log:

Started GET "/jwt/auth?account=machiel.surname%40company_name.com&scope=repository%3Acompany_name%2Fpostfix%2Fimage%3Apush%2Cpull&service=container_registry" for 10.123.12.1 at 2017-06-09 12:30:48 +0200
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"machiel.surname@company_name.com", "scope"=>"repository:company_name/postfix/image:push,pull", "service"=>"container_registry"}
Completed 200 OK in 350ms (Views: 0.4ms | ActiveRecord: 26.7ms)

The registry log is flooded with messages but I don't really see anything wrong

registry_1    | time="2017-06-09T10:32:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=8d47d47a-a638-4d74-bae1-b337e6301383 http.request.method=POST http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=7.90758ms http.response.status=202 http.response.written=0 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "POST /v2/company_name/postfix/image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
registry_1    | time="2017-06-09T10:32:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=018cf528-57b7-42a2-b221-c111c729bced http.request.method=POST http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=10.498925ms http.response.status=202 http.response.written=0 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "POST /v2/company_name/postfix/image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
registry_1    | time="2017-06-09T10:32:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=687076f3-86ea-4be4-8566-56b23a8f0d4e http.request.method=POST http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=8.362401ms http.response.status=202 http.response.written=0 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "POST /v2/company_name/postfix/image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
registry_1    | time="2017-06-09T10:32:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=7a852351-471d-474b-a6eb-223c790cd672 http.request.method=POST http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=21.565296ms http.response.status=202 http.response.written=0 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "POST /v2/company_name/postfix/image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
registry_1    | time="2017-06-09T10:32:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=3e15630f-8dcb-48e4-a0e9-88c06a59d7e8 http.request.method=POST http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=14.294277ms http.response.status=202 http.response.written=0 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "POST /v2/company_name/postfix/image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
registry_1    | time="2017-06-09T10:32:45Z" level=error msg="response completed with error" auth.user.name=msurname err.code="blob unknown" err.detail=sha256:bd97b43c27e332fc4e00edf827bbc26369ad375187ce6eee91c616ad275884b1 err.message="blob unknown to registry" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=2c0574b0-b0b1-4cf0-a58a-109ffb8f86a7 http.request.method=HEAD http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/sha256:bd97b43c27e332fc4e00edf827bbc26369ad375187ce6eee91c616ad275884b1" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.5068ms http.response.status=404 http.response.written=157 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f vars.digest="sha256:bd97b43c27e332fc4e00edf827bbc26369ad375187ce6eee91c616ad275884b1" vars.name="company_name/postfix/image" version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "HEAD /v2/company_name/postfix/image/blobs/sha256:bd97b43c27e332fc4e00edf827bbc26369ad375187ce6eee91c616ad275884b1 HTTP/1.1" 404 157 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
registry_1    | time="2017-06-09T10:32:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="registry.gitlab.example.com:5000" http.request.id=50e3bba5-e2b3-4bc2-9092-f2bbbb21a21b http.request.method=POST http.request.remoteaddr=212.32.23.4 http.request.uri="/v2/company_name/postfix/image/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=7.418354ms http.response.status=202 http.response.written=0 instance.id=0a24dbbf-1c06-4760-8cde-f1081135412f version=v2.6.1 
registry_1    | 10.123.12.1 - - [09/Jun/2017:10:32:45 +0000] "POST /v2/company_name/postfix/image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-79-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"

lotsa empty data files:

root@backend:~/gitlab/gitlab/data/shared/registry/docker/registry/v2/repositories/company_name/postfix/image/_uploads# ls -l ` find .  -iname data ` 
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./14020125-fac8-4bee-aa8c-2a8f7fa63267/data
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./1854d5a3-322a-42c3-9fc7-1d619b78f748/data
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./191ca47b-3d2d-489a-8bfd-681f75597f2e/data
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./19f223a2-bcec-4c4c-9033-f6e0a27f1d7a/data
-rw-r--r-- 1 root root 0 Jun  9 12:01 ./3456c1b3-5253-4799-9609-70495cb1bd2c/data
-rw-r--r-- 1 root root 0 Jun  9 12:01 ./493ff60a-19ed-4006-ad14-f2f86d9cd876/data
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./4bff508a-23e7-4c4f-8c61-d55afc864c56/data
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./4e444d91-63b7-44ad-af6e-f834991cf5e8/data
-rw-r--r-- 1 root root 0 Jun  9 12:00 ./6be7a43c-d8d9-40af-ab81-8d8a4a22825a/data
-rw-r--r-- 1 root root 0 Jun  9 12:01 ./7149d946-8bde-4184-98e5-a9fec64748b5/data

This is my docker-compose.yml:

version: '2'

services:
  redis:
    restart: always
    image: sameersbn/redis:latest
    command:
    - --loglevel warning
    volumes:
    - /root/gitlab/redis/data:/var/lib/redis:Z

  postgresql:
    restart: always
    image: sameersbn/postgresql:9.6-2
    volumes:
    - /root/gitlab/postgresql/data:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS={{ gitlab_db_password }}
    - DB_NAME=company_name_gitlab
    - DB_EXTENSION=pg_trgm

  registry: 
    image: registry:2
    restart: always
    ports: 
      - "{{ internal_ip }}:5000:5000"
    volumes:
      - ./gitlab/shared/registry:/registry
      - ./certs/registry-auth.crt:/certs/registry-auth.crt
    environment:
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
      - REGISTRY_AUTH_TOKEN_REALM=https://gitlab.example.com/jwt/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
      - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry-auth.crt
      - REGISTRY_STORAGE_DELETE_ENABLED=true

  gitlab:
    restart: always
    image: sameersbn/gitlab
    mem_limit: 2g
    links:
      - registry:registry.gitlab.example.com
    depends_on:
    - redis
    - postgresql
    ports:
    - "{{ internal_ip }}:10080:80"
    - "{{ internal_ip }}:10022:22"
    volumes:
    - .//gitlab/data:/home/git/data:Z
    - ./certs/:/certs/
    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS={{ gitlab_db_password }}
    - DB_NAME=company_name_gitlab

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Europe/Amsterdam
    - GITLAB_TIMEZONE=Amsterdam

    - GITLAB_HTTPS=true
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=gitlab.example.com
    - GITLAB_PORT=443
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=
    - GITLAB_SECRETS_DB_KEY_BASE={{gitlab_secret_db_key_base}}
    - GITLAB_SECRETS_SECRET_KEY_BASE={{gitlab_secret_key}}
    - GITLAB_SECRETS_OTP_KEY_BASE={{ gitlab_secret_otp_key_base }}

    - GITLAB_ROOT_PASSWORD={{ gitlab_root_password }}
    - GITLAB_ROOT_EMAIL={{ gitlab_root_email }}

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - GITLAB_EMAIL=notifications@gitlab.{{ mailserver_domain }}
    - GITLAB_EMAIL_REPLY_TO=noreply@gitlab.{{ mailserver_domain }}
    - GITLAB_INCOMING_EMAIL_ADDRESS=reply@gitlab.{{ mailserver_domain }}

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00

    - SMTP_ENABLED=true
    - SMTP_DOMAIN=gitlab.{{ mailserver_domain }}
    - SMTP_HOST={{ mailserver_ip }}
    - SMTP_PORT=25
    - SMTP_USER={{ gitlab_mail_username }}
    - SMTP_PASS={{ gitlab_mail_password }}
    - SMTP_STARTTLS=false
    - SMTP_AUTHENTICATION=login

    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST={{ gitlab_registry_host }}
    - GITLAB_REGISTRY_PORT=5000
    - GITLAB_REGISTRY_API_URL=http://{{ gitlab_registry_host }}:5000
    - GITLAB_REGISTRY_KEY_PATH=/certs/registry-auth.key
    - GITLAB_REGISTRY_ISSUER=gitlab-issuer

    - IMAP_ENABLED=false
    - IMAP_HOST=imap.gmail.com
    - IMAP_PORT=993
    - IMAP_USER=mailer@example.com
    - IMAP_PASS=password
    - IMAP_SSL=true
    - IMAP_STARTTLS=false

    - OAUTH_ENABLED=false
    - OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
    - OAUTH_ALLOW_SSO=
    - OAUTH_BLOCK_AUTO_CREATED_USERS=true
    - OAUTH_AUTO_LINK_LDAP_USER=false
    - OAUTH_AUTO_LINK_SAML_USER=false
    - OAUTH_EXTERNAL_PROVIDERS=

    - OAUTH_CAS3_LABEL=cas3
    - OAUTH_CAS3_SERVER=
    - OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
    - OAUTH_CAS3_LOGIN_URL=/cas/login
    - OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
    - OAUTH_CAS3_LOGOUT_URL=/cas/logout

    - OAUTH_GOOGLE_API_KEY=
    - OAUTH_GOOGLE_APP_SECRET=
    - OAUTH_GOOGLE_RESTRICT_DOMAIN=

    - OAUTH_FACEBOOK_API_KEY=
    - OAUTH_FACEBOOK_APP_SECRET=

    - OAUTH_TWITTER_API_KEY=
    - OAUTH_TWITTER_APP_SECRET=

    - OAUTH_GITHUB_API_KEY=
    - OAUTH_GITHUB_APP_SECRET=
    - OAUTH_GITHUB_URL=
    - OAUTH_GITHUB_VERIFY_SSL=

    - OAUTH_GITLAB_API_KEY=
    - OAUTH_GITLAB_APP_SECRET=

    - OAUTH_BITBUCKET_API_KEY=
    - OAUTH_BITBUCKET_APP_SECRET=

    - OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
    - OAUTH_SAML_IDP_CERT_FINGERPRINT=
    - OAUTH_SAML_IDP_SSO_TARGET_URL=
    - OAUTH_SAML_ISSUER=
    - OAUTH_SAML_LABEL="Our SAML Provider"
    - OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
    - OAUTH_SAML_GROUPS_ATTRIBUTE=
    - OAUTH_SAML_EXTERNAL_GROUPS=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=

    - OAUTH_CROWD_SERVER_URL=
    - OAUTH_CROWD_APP_NAME=
    - OAUTH_CROWD_APP_PASSWORD=

    - OAUTH_AUTH0_CLIENT_ID=
    - OAUTH_AUTH0_CLIENT_SECRET=
    - OAUTH_AUTH0_DOMAIN=

    - OAUTH_AZURE_API_KEY=
    - OAUTH_AZURE_API_SECRET=
    - OAUTH_AZURE_TENANT_ID=
ChessSpider commented 7 years ago

fixed, problem was in my haproxy. Changed some random values after I noticed my client was sending a PATCH request in cleartext isntead of SSL mumble jumble.

Changed it to:

frontend docker-in
    mode tcp
    option httpclose
    bind *:5000 ssl crt {{ cert_path }}
    acl is_registry hdr(host) -i {{ gitlab_registry_host }} 

    reqadd X-Forwarded-Port:\ 5000
    reqadd X-Forwarded-Proto:\ https
    reqadd X-Forwarded-Scheme:\ https
    reqadd X-Forwarded-Host:\ {{ gitlab_registry_host }}

    rsprep ^Location:\ http://(.*) Location:\ https://\1
    redirect scheme https code 301 if !{ ssl_fc }

    use_backend registry if is_registry
    default_backend registry

dont forget to change the {{ varname }} with the correct value

dont ask me why it works or which ones are required and which ones are not... im just happy it works