search

kubernetes-retired / contrib

[EOL] This is a place for various components in the Kubernetes ecosystem that aren't part of the Kubernetes core.
Apache License 2.0
2.46k stars 1.68k forks source link

Nginx-Ingress: Docker Registry behind Ingress - docker push don't work #2444

Closed muffin87 closed 7 years ago

muffin87 commented 7 years ago

Hi everyone,

im running K8s version 1.5.2 on CentOS. I wanted to deploy a vanilla docker registry and expose the service with Nginx Ingress controller.

Issue

docker push does not work with the nginx-ingress-controller in front of the docker-registry.

It seems to me that something in the nginx-ingress-controller config is blocking the docker push payload upload (tar file).

Any idea whats happening here and how I can solve this?

Error Message:

docker-client: Error: Status 404 trying to push repository nginx: "404 page not found\n"

dockerd Upload failed: EOF

Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.177922934Z" level=debug msg="Pushing layer: sha256:809c8c0dd73c9081a8ab9795b5fb9beab60c7f602a0886dbbdf48e939d608163"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.401819584Z" level=debug msg="Assembling tar data for d388be1b57d4bbb37c16836b3fafba89e433a413f7448680d6992ecfe4f07089"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.416382683Z" level=debug msg="Assembling tar data for 6780c17ba8c81862656e46d0812c0eaf7fc2eb55447cf1f71b1e3e64e548256b"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.416555479Z" level=debug msg="Assembling tar data for 6c84bf0c3c99e5454f5536068db727174c4dea0add6c7c54052604dbbba4a46a"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.519050420Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.537034379Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:29:51 moby root: time="2017-03-03T11:29:51.541124148Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:29:53 moby root: time="2017-03-03T11:29:53.983465337Z" level=debug msg="Pushing layer: sha256:e8fa134cb7b8a5e2943c9ea6e48a90c9f7fc80fd26780ce9d74bbab927a6cfc5"
Mar  3 11:29:54 moby root: time="2017-03-03T11:29:54.227004087Z" level=debug msg="Assembling tar data for b3c45395c4b867925ed531458101d8c5cb8c5462f06a678c8132f06e2db8c588"
Mar  3 11:29:54 moby root: time="2017-03-03T11:29:54.345157833Z" level=error msg="Upload failed, retrying: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.520650325Z" level=debug msg="Pushing layer: sha256:7cbcbac42c44c6c38559e5df3a494f44987333c8023a40fec48df2fce1fc146b"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.537773800Z" level=debug msg="Pushing layer: sha256:e8d45b8ab3ca8a83ba4e340f4b901c593da00f7f2bb068937a8b32b3beb606af"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.541707653Z" level=debug msg="Pushing layer: sha256:809c8c0dd73c9081a8ab9795b5fb9beab60c7f602a0886dbbdf48e939d608163"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.744771310Z" level=debug msg="Assembling tar data for 6c84bf0c3c99e5454f5536068db727174c4dea0add6c7c54052604dbbba4a46a"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.756433036Z" level=debug msg="Assembling tar data for 6780c17ba8c81862656e46d0812c0eaf7fc2eb55447cf1f71b1e3e64e548256b"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.760178212Z" level=debug msg="Assembling tar data for d388be1b57d4bbb37c16836b3fafba89e433a413f7448680d6992ecfe4f07089"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.851262730Z" level=error msg="Upload failed: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.856225668Z" level=error msg="Upload failed: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.863372727Z" level=error msg="Upload failed: EOF"
Mar  3 11:30:11 moby root: time="2017-03-03T11:30:11.863480799Z" level=error msg="Attempting next endpoint for push after error: EOF"

nginx-ingress log:

<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] "HEAD /v2/nginx/blobs/sha256:ba1ed986f6a0df1b3fa02fe8cf76e45e7f7ae248c2a2d9f6ce901585c38d5a79 HTTP/1.1" 404 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:20 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:21 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"
<censored> - ucpapp [03/Mar/2017:11:29:21 +0000] "HEAD /v2/nginx/blobs/sha256:a362ec94dcabb258895c0cde3905dafd6c40df5ffa4a668fdf6b175655774192 HTTP/1.1" 404 0 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" "-"

Setup

dockerd -> nginx-ingress -> k8s-service-object -> nginx-for-auth -> registry

Configs

rendered nginx-config

upstream tools-docker-registry-registry.<censored>-docker-registry {

  server 10.254.4.9:80;
}

server {
  listen 80;

  server_name registry.<censored>;

  location / {
    proxy_http_version 1.1;

    proxy_connect_timeout 60s;
    proxy_read_timeout 60s;
    client_max_body_size 1m;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering on;

    proxy_pass http://tools-docker-registry-registry.<censored>-docker-registry;

  }
}

registry ingress conf

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: docker-registry
  namespace: tools
spec:
  rules:
    - host: registry.<censored>
      http:
        paths:
        - backend:
            serviceName: docker-registry
            servicePort: 80
aledbf commented 7 years ago

@muffin87 what ingress controller and version are you using?

muffin87 commented 7 years ago

@aledbf: I tried several versions:

I also tried the other controller from https://github.com/nginxinc/kubernetes-ingress

aledbf commented 7 years ago

@muffin87 the nginx-config is from your nginx right?

Add the annotation ingress.kubernetes.io/proxy-body-size: 0 to the ingress rule. By default this value is 1m.

aledbf commented 7 years ago

@muffin87 in the ingress rule the service docker-registry is the real docker registry or your nginx?

andiMenge commented 7 years ago

@aledbf docker-registry in the ingress rule points to the nginx used for authentication, not the registry itself.

ingress.kubernetes.io/proxy-body-size: 0 has no effect on the issue

aledbf commented 7 years ago

@andiMenge please switch to the real docker registry to verify you can push an image (test the annotation is working)

andiMenge commented 7 years ago

@aledbf We tried this, with the same results. Actually we just implemented the nginx-for-auth just to see if it makes a difference.

If I bypass the ingress-controller and push directly to the k8s-service-object via NodePort everthing works fine

aledbf commented 7 years ago

@andiMenge please post the output of curl -v http://registry.<censored>/v2/, the nginx.conf from the ingress controller using kubectl exec <ingress pod> cat /etc/nginx/nginx.conf and the logskubectl logs <ingress pod>.

andiMenge commented 7 years ago

@aledbf I switched back to our old and simpler config. The setup looks now like this:

dockerd -> nignx-ingress-controller -> k8s-registry-service-object -> registry

nginx-ingress-controller:0.9.0-beta.1

ingress-controller log

::ffff:<censored> - [::ffff:<censored>] - ucpapp [03/Mar/2017:14:36:30 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "curl/7.51.0" 133 0.003 [tools-docker-registry-5000] 10.254.31.12:5000 2 0.003 200

curl -vkL -u user:pw http://registry.censored/v2/

*   Trying <censored>...
* TCP_NODELAY set
* Connected to registry.<censored> (<censored>) port 80 (#0)
* Server auth using Basic with user '<censored>'
> GET /v2/ HTTP/1.1
> Host: registry.<censored>
> Authorization: Basic dWNwYXBwOiwuYXBwLHVjcA==
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.11.10
< Date: Fri, 03 Mar 2017 14:36:30 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
<
* Curl_http_done: called premature == 0
* Connection #0 to host registry.<censored> left intact
{}%

ingress-controller config

daemon off;

worker_processes 64;
pid /run/nginx.pid;
events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;
    set_real_ip_from    0.0.0.0/0;
    real_ip_recursive   on;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;
    # lua section to return proper error codes when custom pages are used
    lua_package_path '.?.lua;./etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
    init_by_lua_block {
        require("error_page")
    }

    sendfile            on;
    aio                 threads;
    tcp_nopush          on;
    tcp_nodelay         on;
    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout 75s;

    client_header_buffer_size       1k;
    large_client_header_buffers     4 8k;

    types_hash_max_size             2048;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;

    include /etc/nginx/mime.types;
    default_type text/html;
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;

    server_tokens on;

    log_format upstreaminfo '$remote_addr - '
        '[$proxy_add_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" '
        '$request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {
        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
    error_log  /var/log/nginx/error.log notice;

    resolver 10.253.0.8 valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
       default           $http_x_forwarded_port;
       ''                $server_port;
    }

    # map port 442 to 443 for header X-Forwarded-Port
    map $pass_server_port $pass_port {
        442              443;
        default          $pass_server_port;
    }

    # Map a response error watching the header Content-Type
    map $http_accept $httpAccept {
        default          html;
        application/json json;
        application/xml  xml;
        text/plain       text;
    }

    map $httpAccept $httpReturnType {
        default          text/html;
        json             application/json;
        xml              application/xml;
        text             text/plain;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    # turn on session caching to drastically improve performance
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    # In case of errors try the next upstream server before returning an error
    proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

    upstream tools-docker-registry-5000 {
        least_conn;
        server 10.254.31.12:5000 max_fails=0 fail_timeout=0;
    }

    upstream upstream-default-backend {
        least_conn;
        server 10.254.31.11:8080 max_fails=0 fail_timeout=0;
    }
    server {
        server_name _;
        listen [::]:80 ipv6only=off default_server reuseport backlog=511;
        listen 442  default_server reuseport backlog=511 ssl http2;
        # PEM sha: 146d6a1a176900b4af00faf1b177c82d6c7b39ae
        ssl_certificate                         /ingress-controller/ssl/system-snake-oil-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/system-snake-oil-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains; preload";
        location / {
            set $proxy_upstream_name "upstream-default-backend";

            port_in_redirect off;
            client_max_body_size                    "1m";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
            proxy_pass http://upstream-default-backend;
        }
        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }
    }

    server {
        server_name registry.<censored>;
        listen [::]:80;

        location / {
            set $proxy_upstream_name "tools-docker-registry-5000";

            port_in_redirect off;

            auth_basic "Authentication Required - Docker-Registry";
            auth_basic_user_file /etc/ingress-controller/auth/tools-docker-registry.passwd;
            proxy_set_header Authorization "";

            client_max_body_size                    "0";

            proxy_set_header Host                   $host;

            # Pass Real IP
            proxy_set_header X-Real-IP              $remote_addr;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";

            proxy_http_version                      1.1;
            proxy_pass http://tools-docker-registry-5000;
        }
    }

    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx/command.go#L104
        listen [::]:18080 ipv6only=off default_server reuseport backlog=511;

        location /healthz {
            access_log off;
            return 200;
        }
        location /nginx_status {
            access_log off;
            stub_status on;
        }

        # this location is used to extract nginx metrics
        # using prometheus.
        # TODO: enable extraction for vts module.
        location /internal_nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }

        location / {
            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass             http://upstream-default-backend;
        }

    }

    # default server for services without endpoints
    server {
        listen 8181;
        set $proxy_upstream_name "-";

        location / {
            return 503;
        }
    }
}

stream {
    # map FQDN that requires SSL passthrough
    map $ssl_preread_server_name $stream_upstream {
        # send SSL traffic to this nginx in a different port
        default                         nginx-ssl-backend;
    }

    log_format log_stream '$remote_addr [$time_local] $protocol [$ssl_preread_server_name] [$stream_upstream] $status $bytes_sent $bytes_received $session_time';

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # configure default backend for SSL
    upstream nginx-ssl-backend {
        server 127.0.0.1:442;
    }
    server {
        listen                  [::]:443 ipv6only=off;
        proxy_pass              $stream_upstream;
        ssl_preread             on;
    }

    # TCP services

    # UDP services
}

ingress.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: docker-registry
  namespace: tools
  annotations:
    # type of authentication
    ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    ingress.kubernetes.io/auth-secret: docker-registry-basic-auth
    # message to display with an appropiate context why the authentication is required
    ingress.kubernetes.io/auth-realm: "Authentication Required - Docker-Registry"
    # request body maximum allowed size unlimited
    ingress.kubernetes.io/proxy-body-size: "0"
spec:
  rules:
    - host: registry.<censored>
      http:
        paths:
        - backend:
            serviceName: docker-registry
            servicePort: 5000

service.yml

apiVersion: v1
kind: Service
metadata:
  name: docker-registry
  namespace: tools
spec:
  selector:
    app: docker-registry
  type: NodePort
  ports:
  - port: 5000
    protocol: TCP

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: docker-registry
  namespace: tools
  labels:
    app: docker-registry
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
        - name: docker-registry
          image: registry:2.6
          ports:
            - containerPort: 5000
        #   volumeMounts:
        #     - mountPath: /var/lib/registry
        #       name: image-data-volume
        # volumes:
        #   - name: image-data-volume
        #     hostPath:
        #       path: /data/docker-registry
aledbf commented 7 years ago

@muffin87 from the log and the curl output this is working. Did you configured your docker client to use registry.<censored> as an insecure registry?

andiMenge commented 7 years ago

@aledbf in this case: yes, registry.<censored> is configured insecure. Normally we have a F5 Loadbalancer in front of the ingress-controller that handles TSL termination.

For debugging reasons I am bypassing the F5 so I have to use the insecure registry. Connecting through the F5 results in the same error.

andiMenge commented 7 years ago

@aledbf this is what we can see in the ingress-controller logs while doing a docker push

::ffff:<censored> - [::ffff:<censored>] - ucpapp [03/Mar/2017:15:27:54 +0000] "PUT /v1/repositories/openjdk/ HTTP/1.1" 404 19 "-" "docker/17.03.0-ce go/go1.7.5 git-commit/3a232c8 kernel/4.9.12-moby os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.0-ce \x5C(darwin\x5C))" 623 0.041 [tools-docker-registry-5000] 10.254.31.12:5000 19 0.000 404

I am not sure how a docker pushworks under the hood and why dockerd is switching to /v1/.

aledbf commented 7 years ago

@andiMenge please check the docker-registry logs. I think you will find the same 404 error. If that is the case then the registry is returning the error, not the ingress controller. Can you pull from the registry?

andiMenge commented 7 years ago

@aledbf you are right, the 404 is from the registry. Strange thing is, if I bypass the ingress, push and pulls work.

This is what I get if I try to pull (currently there is no valid image inside the registry)

Mar  3 15:59:50 moby root: time="2017-03-03T15:59:50.085761251Z" level=debug msg="[registry] Calling GET https://registry.<censored>/v1/repositories/openjdk/images"
Mar  3 15:59:50 moby root: time="2017-03-03T15:59:50.304449080Z" level=error msg="Not continuing with pull after error: Error: image openjdk:latest not found"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.656002530Z" level=debug msg="Calling GET /_ping"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.698394305Z" level=debug msg="Calling POST /v1.26/images/create?fromImage=registry.<censored>%2Fopenjdk&tag=latest"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.779542000Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.<censored>"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.819091547Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.<censored>"
Mar  3 16:02:08 moby root: time="2017-03-03T16:02:08.819317223Z" level=debug msg="Trying to pull registry.<censored>/openjdk from https://registry.<censored> v2"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.312269446Z" level=error msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.312431155Z" level=debug msg="Trying to pull registry.<censored>/openjdk from https://registry.<censored> v1"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.352921913Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.<censored>"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.353219839Z" level=debug msg="attempting v1 ping for registry endpoint https://registry.<censored>/v1/"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582216052Z" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582360892Z" level=debug msg="PingResult.Version: \"\""
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582427266Z" level=debug msg="Registry standalone header: ''"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582488624Z" level=debug msg="PingResult.Standalone: true"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582552560Z" level=debug msg="Endpoint https://registry.<censored>/v1/ is eligible for private registry. Enabling decorator."
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.582660693Z" level=debug msg="[registry] Calling GET https://registry.<censored>/v1/repositories/openjdk/images"
Mar  3 16:02:09 moby root: time="2017-03-03T16:02:09.817444263Z" level=error msg="Not continuing with pull after error: Error: image openjdk:latest not found"

I will do some tests with valid images inside the registry later...

andiMenge commented 7 years ago

This is what it looks like from inside the registry:

my earlier push of the openjdk image was recognised...

/var/lib/registry/docker/registry/v2/repositories/openjdk/_uploads # ls
25da551f-cd87-45bc-8489-edd1c5523a1f  87b2ece7-84e7-4c9e-84d0-7409b133dab2
32e6fdec-31e8-47ba-a3c5-fb029485dcf8  8b35825c-7514-4f29-9c68-103d5913bd37
336c4928-5060-49d5-8baf-26407965e8ed  8dfb236f-3bd5-4261-9980-66a6aad1dbaa
39a4d3a2-32d6-42ef-9239-2a55243854b5  a9105502-e429-406c-8a6b-d598e0150845
3da96d01-a6f3-4141-a3bc-dcb5c4b2ef39  b08bcd17-19ce-424d-aa50-7202c647c501
3f647174-3a9b-4942-b12e-a88bd47127cc  b155623f-e2ac-4812-8e7c-db22c5b447c3
4585d21d-2414-4d57-afda-999dc56b212d  b17045ae-e498-47c0-9b69-9041b636ec62
48632729-5f29-4deb-9975-3e157a38ad70  b66e2ac1-f86e-48dc-8b82-0540f9cbcb5f
491cf7c4-d9ad-40f6-87d3-aaf7c509b7a8  bb6dc397-f344-43fb-999e-a66c572a80eb
4c40f4de-0b8b-4168-8ae1-9103bffdc898  c26de260-d9d8-447b-9693-ef0864af258a
5c666d9f-d074-48d4-9f5f-47f7efbcea9e  c52b329a-c573-4b5d-a2b5-3b727da3a9a1
611794f5-0e09-411f-8fa9-8a00118d55c1  c8dc6bea-98ed-4903-ad39-7cf220b04f29
673572fd-4082-43fd-9b5d-8a5418e1adf8  cb4676b1-69f5-478f-a205-bb250d803272
6caada73-951c-4403-91e9-9f0fa5c8eb8e  fc317e83-4971-4003-8813-771b72a73c86
6cefc76f-f979-4d79-b41e-2ca335de916a  fc3df036-3c9e-4395-b8c9-dcad0c1bd339
/var/lib/registry/docker/registry/v2/repositories/openjdk/_uploads # du -sch
244.0K  .
244.0K  total
andiMenge commented 7 years ago

the issue was a misconfiguration in the registry that caused redirecting issues in the ingress

if set properly, the ingress works fine host: https://myregistryaddress.org:5000

mtrofimm commented 7 years ago

@andiMenge I have the same issue. Can you give some more information how you solved this problem?

muffin87 commented 7 years ago

@mtrofimm:

You have to set the right URL in your registry config (config.yml).

Example: Your registry should be reachable under https://my-awesome-registry.myproject.de

Registry config:

    http:
      addr: :5000
      debug:
        addr: :5001
      host: https://my-awesome-registry.myproject.de
      headers:
        X-Content-Type-Options: [nosniff]

And you also have to set this URL in your ingress manifest.

K8s ingress file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-registry
  namespace: mynamespace
  annotations:
    # type of authentication
    ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    ingress.kubernetes.io/auth-secret: docker-registry-basic-auth
    # message to display with an appropiate context why the authentication is required
    ingress.kubernetes.io/auth-realm: "Authentication Required - Docker-Registry"
    # request body maximum allowed size unlimited
    ingress.kubernetes.io/proxy-body-size: "0"
spec:
  rules:
    - host: my-awesome-registry.myproject.de
      http:
        paths:
        - backend:
            serviceName: my-registry
            servicePort: 5000
shenshouer commented 6 years ago

I push docker image to registry behind nginx ingress and meet the error: blob unknown to registry.

aledbf commented 6 years ago

@muffin87 @shenshouer I will add an example to show how is possible to expose a docker registry running in the cluster using kube-lego without limit in the uploads.

shenshouer commented 6 years ago

I think the ingress.kubernetes.io/proxy-body-size: "0" int annotations of nginx ingress doesn`t work fine!

angapov commented 6 years ago

It's nginx.ingress.kubernetes.io/proxy-body-size: "0" now