Error: 413 "Request Entity Too Large" with skopeo

PDVJAM commented 12 months ago

Hey. I have a private docker registry with nginx in front. If I do docker pull, tag, and push to this registry - everything works without problems. But If I do the same with skopeo - it gives me 413 errors in the end. Please, suggest, what might be wrong.

I have added client_max_body_size 0; into nginx configuration everywhere, but no changes. So seems something with the skopeo.

My nginx:

server {
        listen 443 ssl http2;

        root /var/www/registry/html;
        index index.html index.htm index.nginx-debian.html;

        server_name registry.com;
        client_max_body_size 0;

    ssl_certificate /etc/letsencrypt/live/registry.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/registry.com/privkey.pem;

#    include /etc/nginx/conf.d/options-ssl-nginx.conf;

    location ~ /.well-known/acme-challenge {
            allow all;
            root /var/www/registry/html;
    }

location / {
    client_max_body_size 0;
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    proxy_pass                          http://localhost:5000;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
        }
}

Docker compose for registry:

version: '3'

services:
  registry:
    image: registry:latest
    restart: always
    ports:
    - "5000:5000"
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
    volumes:
      - ./data:/data
      - ./auth:/auth

Ordinary docker pull,tag,pull:

root@docker-sync:~/docker-registry# docker pull registry.digitalocean.com/.../fe:v15
v15: Pulling from .../fe
...
Status: Downloaded newer image for registry.digitalocean.com/.../fe:v15

root@docker-sync:~/docker-registry# docker tag 24c4c06179a0 registry.com/.../fe:v15

root@docker-sync:~/docker-registry# docker push registry.com/.../fe:v15
The push refers to repository [registry.com/.../fe]
...
4693057ce236: Pushed
v15: digest: sha256:8811971cdf5d01212781372412f93ac0b1701709633f82416c7934493a2fdad6 size: 3459

And with the skopeo:


root@docker-sync:~/docker-registry# skopeo copy --src-creds user:pw docker://registry.digitalocean.com/.../fe:v15 --dest-creds user:pw docker://registry.com/.../fe:v15
...
Copying blob 14ba83f20a37 [--------------------------------------] 47.5KiB / 193.8MiB
Copying blob b948fc8cfd29 done
Copying blob 9d80b45a59bc done
Copying blob 97ae215797dd done
Copying blob f6d1950b524d done
FATA[0006] writing blob: uploading layer chunked: StatusCode: 413, <html>
<head><title>413 Request Entity Too Large</...

PDVJAM commented 12 months ago

Yeah, that is definitely something with the nginx, because if I do registry wo proxy - all works:

INFO[0053] Copying image ref 4/5                         from="docker://registry.digitalocean.com/.../fe:v3" to="docker://127.0.0.1:443/v2/fe:v3"
Getting image source signatures
Copying blob d68f2f1a5d31 [--------------------------------------] 0.0b / 0.0b
Copying blob 904e9a84b7a0 skipped: already exists
Copying blob e7e6b7606c1a skipped: already exists
Copying blob e2c28eeda74b skipped: already exists
Copying blob 7264a8db6415 skipped: already exists
Copying blob 681a0b221c96 done
Copying blob 62879395397a done
Copying blob e7b2b4c12840 done
Copying blob cda0560b0ea5 done
Copying blob d06a03355179 done
Copying blob 5a570be44733 done
Copying blob 59b05be6bfb6 done
Copying blob 13b73343cc72 done
Copying config b0372eb7cb done
Writing manifest to image destination
Storing signatures

Kinda strange.

mtrmac commented 12 months ago

Thanks for your report.

Does timing make a difference? If you skopeo copy docker://$source dir:$tmp && skopeo copy dir:$tmp docker://$dest, does that work?
If I had to guess, one difference probably is that skopeo copy sets a Content-Length header in such a copy, while docker push does an on-line compression and quite possibly doesn’t set one.
I see a http_user_agent condition in the quoted config… it is very tempting to me to assume that there might be some other User-Agent-based condition that lets Docker through.

PDVJAM commented 12 months ago

Does timing make a difference? If you skopeo copy docker://$source dir:$tmp && skopeo copy dir:$tmp docker://$dest, does that work?

Yep, it is the same.

I see a http_user_agent condition in the quoted config… it is very tempting to me to assume that there might be some other User-Agent-based condition that lets Docker through.

Tried to remove this block, but it is the same.

Thank you. Since it works with the direct connection - I can finish my task. Probably later will try to investigate more, but not now.

containers / skopeo

Error: 413 "Request Entity Too Large" with skopeo #2104