search

docker-archive / docker-registry

This is **DEPRECATED**! Please go to https://github.com/docker/distribution
Apache License 2.0
2.88k stars 879 forks source link

docker pull fails with private registry #253

Open strcrzy opened 10 years ago

strcrzy commented 10 years ago

i am able to start the registry, and push to it, but pulling seems to fail miserably. when both the registry and docker are hosted on AWS, things hang at the Pulling fs layer part. when only the registry is hosted on AWS and docker is not, docker is able to pull from the registry, but at very slow speeds (20k/s).

i am able to push to the registry from both of these locations without incident the registry config, and logs for one of the pulls are below. i have tried using both the prod and dev flavors, with no luck with either. i feel like i'm missing something, but i can't seem to zero in on it.

common:
    loglevel: _env:LOG_LEVEL
    secret_key: _env:REGISTRY_SECRET
    standalone: true
    disable_token_auth: true

prod:
    storage: s3
    boto_bucket: _env:AWS_BUCKET
    s3_access_key: _env:AWS_KEY
    s3_secret_key: _env:AWS_SECRET
    s3_bucket: _env:AWS_BUCKET
    s3_encrypt: true
    s3_secure: true
    s3_encrypt: true
    s3_secure: true
    storage_path: /images/

local:
    storage: local
    storage_path: _env:REGISTRY_ROOT

dev:
    storage: local
    storage_path: /tmp/registry/
    loglevel: debug
63.166.156.188 - - [19/Feb/2014:00:32:17] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
2014-02-19 00:32:17,718 INFO: 63.166.156.188 - - [19/Feb/2014:00:32:17] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
2014-02-19 00:32:18,218 DEBUG: check_session: Session is empty
63.166.156.188 - - [19/Feb/2014:00:32:18] "GET /v1/repositories/troupe/images HTTP/1.1" 200 93 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:18,219 INFO: 63.166.156.188 - - [19/Feb/2014:00:32:18] "GET /v1/repositories/troupe/images HTTP/1.1" 200 93 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:19,057 DEBUG: check_session: Session is empty
2014-02-19 00:32:19,057 DEBUG: [get_tags] namespace=library; repository=troupe
63.166.156.188 - - [19/Feb/2014:00:32:19] "GET /v1/repositories/library/troupe/tags HTTP/1.1" 200 84 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:19,058 INFO: 63.166.156.188 - - [19/Feb/2014:00:32:19] "GET /v1/repositories/library/troupe/tags HTTP/1.1" 200 84 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:19,553 DEBUG: check_session: Session is empty
63.166.156.188 - - [19/Feb/2014:00:32:19] "GET /v1/images/5d1541904627057ccb4ab7f9e22a9456bca29a6cdd0bcdee9f731b593d7f80d8/ancestry HTTP/1.1" 200 74 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:19,554 INFO: 63.166.156.188 - - [19/Feb/2014:00:32:19] "GET /v1/images/5d1541904627057ccb4ab7f9e22a9456bca29a6cdd0bcdee9f731b593d7f80d8/ancestry HTTP/1.1" 200 74 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:20,058 DEBUG: check_session: Session is empty
63.166.156.188 - - [19/Feb/2014:00:32:20] "GET /v1/images/5d1541904627057ccb4ab7f9e22a9456bca29a6cdd0bcdee9f731b593d7f80d8/json HTTP/1.1" 200 619 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:20,059 INFO: 63.166.156.188 - - [19/Feb/2014:00:32:20] "GET /v1/images/5d1541904627057ccb4ab7f9e22a9456bca29a6cdd0bcdee9f731b593d7f80d8/json HTTP/1.1" 200 619 "-" "docker/0.8.0 go/go1.2 git-commit/cc3a8c8 kernel/3.8.0-29-generic os/linux arch/amd64"
2014-02-19 00:32:20,558 DEBUG: check_session: Session is empty
wking commented 10 years ago

On Tue, Feb 18, 2014 at 04:40:18PM -0800, jamie brim wrote:

when only the registry is hosted on AWS and docker is not, docker is able to pull from the registry, but at very slow speeds (20k/s).

I don't know much about Flask, but it would be nice to use sendfile to stream image layers if the kernel supports it. Unfortunately, native sendfile support only landed in Python 3.3 1. It looks like there is some Flask-side support though 2, so maybe it wouldn't be too difficult to do. The current storage.local.LocalStorage.stream_read reads the file into local memory before passing it on to Flask, which seems inefficient.

samalba commented 10 years ago

@wking sendfile is perfect when you deliver your object from a local storage (from the local disk for instance). However since docker-registry supports a set of storage backend (s3, google storage, etc...), objects are served from those remote sources. Sendfile is useless unless we would buffer on the disk first.

To answer @strcrzy problem, I think it looks like a configuration issues. Have you tried with the registry container[1]?

If you use s3, I recommend configuring a LRU cache[2] on the front, it will cache most of the calls to S3.

[1] https://index.docker.io/_/registry/ [2] https://github.com/dotcloud/docker-registry#performance-on-prod

strcrzy commented 10 years ago

@samalba - thanks for the suggestion!

i haven't tried the registry container, how would i configure it? there doesn't seem to be a default config.yml bundled with it.

the only way i could figure out to bundle my own config was to build a container for myself with the dockerfile. is there a config provided at some later stage somehow that i am interfering with?

wking commented 10 years ago

On Wed, Feb 19, 2014 at 10:26:00AM -0800, jamie brim wrote:

i haven't tried the registry container, how would i configure it? there doesn't seem to be a default config.yml bundled with it.

It should use config_sample.yml because of this Dockerfile line 1. You can use environment variable to tune the config to match your local setup.

strcrzy commented 10 years ago

thank you that makes sense now. so switching to the registry container and passing in env vars appropriately, including redis vars, does fix my speed issue pulling remotely, but i still have a problem with pulling from AWS machines. it does not appear to be a connectivity issue, as both pushing and the GET requests for both the /v1/_ping and the /v1/images calls make it to the server, but then it just hangs again.

i'm at a loss really, any more hints would be greatly appreciated

samalba commented 10 years ago

Have you tried the prod setup on AWS + S3? After you set the right config fields, you can switch the registry to prod flavor by setting the env var SETTINGS_FLAVOR=prod before starting the registry. If the registry runs in a docker container, you can specify the env var using the docker cli as an argument to 'run'.

Note that you can also use the container and provide your own config by mounting the config directory in a volume.

strcrzy commented 10 years ago

this is how i currently invoke the registry container:

docker run \
-e STORAGE_PATH=/registry \
-e SETTINGS_FLAVOR=prod \
-e AWS_ACCESS_KEY_ID=<key> \
-e AWS_SECRET_KEY=<secret> \
-e S3_BUCKET=sra-registry \
-e CACHE_REDIS_HOST=172.17.42.1 \
-e CACHE_LRU_REDIS_HOST=172.17.42.1 \
-e CACHE_REDIS_PORT=6379 \
-e CACHE_LRU_REDIS_PORT=6379 \
-e CACHE_REDIS_PASSWORD=password \
-e CACHE_LRU_REDIS_PASSWORD=password \
-p 5000:5000 
registry

this configuration still stalls on docker pull unfortunately :(