docker registry doesn't run out of the box

obeleh commented 10 years ago

sjuul@core001 ~ $ docker run -i -t -p 10.240.168.100:5000:5000 -e GCS_BUCKET=osdocker google/docker-registry
2014-06-03 07:14:43 [1] [INFO] Starting gunicorn 0.14.5
2014-06-03 07:14:43 [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
2014-06-03 07:14:43 [1] [INFO] Using worker: gevent
2014-06-03 07:14:43 [12] [INFO] Booting worker with pid: 12
2014-06-03 07:14:48 [12] [INFO] Worker exiting (pid: 12)
2014-06-03 07:14:48,062 INFO: Worker exiting (pid: 12)
2014-06-03 07:14:49 [1] [INFO] Shutting down: Master
2014-06-03 07:14:49 [1] [INFO] Reason: Worker failed to boot.

I'm going to add more info as I'm working through the steps needed to get it to run

obeleh commented 10 years ago

So I've enabled logging in gunicorn (in run.sh):

exec gunicorn --log-level debug --access-logfile - --debug --max-requests 100 --graceful-timeout $GUNICORN_GRACEFUL_TIMEOUT -t $GUNICORN_SILENT_TIMEOUT -k gevent -b 0.0.0.0:$REGISTRY_PORT -w $GUNICORN_WORKERS wsgi:application

And that results in this output:

2014-06-03 07:19:02,439 DEBUG: Exception in worker process:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
    __import__(module)
  File "/docker-registry/wsgi.py", line 11, in <module>
    import registry
  File "/docker-registry/registry/__init__.py", line 5, in <module>
    from .tags import *
  File "/docker-registry/registry/tags.py", line 18, in <module>
    store = storage.load()
  File "/docker-registry/lib/storage/__init__.py", line 152, in load
    store = gcs.GSStorage(cfg)
  File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
    BotoStorage.__init__(self, config)
  File "/docker-registry/lib/storage/boto_base.py", line 105, in __init__
    self._boto_conn = self.makeConnection()
  File "/docker-registry/lib/storage/gcs.py", line 34, in makeConnection
    return uri.connect()
  File "/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 140, in connect
    **connection_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 47, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 190, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 558, in __init__
    profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 197, in __init__
    self.get_credentials(access_key, secret_key, security_token, profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 268, in get_credentials
    if profile_name is None and profile_name_name.upper() in os.environ:
AttributeError: 'NoneType' object has no attribute 'upper'

obeleh commented 10 years ago

Ah I remember. Boto issue: https://github.com/boto/boto/pull/2302 pip install -U boto fixes this. Probably the docker image needs this as well.

obeleh commented 10 years ago

Which brings me to the next problem:

2014-06-03 07:29:07,878 DEBUG: Exception in worker process:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
    __import__(module)
  File "/docker-registry/wsgi.py", line 11, in <module>
    import registry
  File "/docker-registry/registry/__init__.py", line 5, in <module>
    from .tags import *
  File "/docker-registry/registry/tags.py", line 18, in <module>
    store = storage.load()
  File "/docker-registry/lib/storage/__init__.py", line 152, in load
    store = gcs.GSStorage(cfg)
  File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
    BotoStorage.__init__(self, config)
  File "/docker-registry/lib/storage/boto_base.py", line 105, in __init__
    self._boto_conn = self.makeConnection()
  File "/docker-registry/lib/storage/gcs.py", line 34, in makeConnection
    return uri.connect()
  File "/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 140, in connect
    **connection_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 47, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 190, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 572, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/local/lib/python2.7/dist-packages/boto/auth.py", line 883, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 3 handlers were checked. ['HmacAuthV1Handler', 'OAuth2Auth', 'OAuth2ServiceAccountAuth'] Check your credentials

obeleh commented 10 years ago

As a quick patch I fix the oath2 plugin since we're going to need to connect to the metadata server anyway.

vim /usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py

And change this:

META_TOKEN_URI = ('http://metadata/computeMetadata/v1/instance/'
                    'service-accounts/default/token')

To this:

META_TOKEN_URI = ('http://169.254.169.254/computeMetadata/v1/instance/'
                    'service-accounts/default/token')

obeleh commented 10 years ago

So now the boto library is skipping the google provider because it requires a service account.

Quick hack to make it work. I have no idea how and where I should add this. Editing config.yml didn't work. File: /usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_plugin.py

    if (provider.name == 'google'):
      config.add_section('GoogleCompute')
      config.set('GoogleCompute', 'service_account', 'XXXXXX@developer.gserviceaccount.com')

obeleh commented 10 years ago

Apparently I forgot to save /usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py and I got this message:

2014-06-03 08:39:14,995 DEBUG: Exception in worker process:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
    __import__(module)
  File "/docker-registry/wsgi.py", line 11, in <module>
    import registry
  File "/docker-registry/registry/__init__.py", line 5, in <module>
    from .tags import *
  File "/docker-registry/registry/tags.py", line 18, in <module>
    store = storage.load()
  File "/docker-registry/lib/storage/__init__.py", line 152, in load
    store = gcs.GSStorage(cfg)
  File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
    BotoStorage.__init__(self, config)
  File "/docker-registry/lib/storage/boto_base.py", line 107, in __init__
    self._config.boto_bucket)
  File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 128, in get_bucket
    bucket.get_all_keys(headers, maxkeys=0)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 471, in get_all_keys
    '', headers, **params)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 399, in _get_all
    query_args=query_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 908, in _mexe
    request.authorize(connection=self)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 379, in authorize
    connection._auth_handler.add_auth(self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_plugin.py", line 46, in add_auth
    self.oauth2_client.GetAuthorizationHeader()
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 344, in GetAuthorizationHeader
    return 'Bearer %s' % self.GetAccessToken().token
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 315, in GetAccessToken
    access_token = self.FetchAccessToken()
  File "/usr/local/lib/python2.7/dist-packages/retry_decorator/retry_decorator.py", line 20, in f_retry
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 520, in FetchAccessToken
    raise GsAccessTokenRefreshError()
TypeError: __init__() takes exactly 2 arguments (1 given)

So change this (/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py):

except Exception:
      raise GsAccessTokenRefreshError()

into this:

except Exception as ex:
      raise GsAccessTokenRefreshError(str(ex))

And then got the correct error message:

GsAccessTokenRefreshError: Unable to find the server at metadata

After setting the metadata server to an IP adress it worked.

obeleh commented 10 years ago

Up to now I've hardcoded my GCS_BUCKET and SECRET_KEY into config.yml. Time to set it back to original and try to run:

As described on the main page but with two changes in order to make it work.

Binding to 0.0.0.0 doesn't get you to the outside world. If you want to connect to your repo, bind it to your internal IP and add a firewall rule.
Dont forget to add your secret key, just the bucket isn't enough

docker run -i -t -p 10.240.168.100:5000:5000 -e SECRET_KEY=XXXXXX -e GCS_BUCKET=osdocker google/docker-registry

So I suppose adding secret_key and hardcoding my service account shouldn't be done. I suppose this should have been discovered in some manner. I don't know how. All I know is, without these settings it won't work.

proppy commented 10 years ago

Are you trying to get the registry running on GCE or somewhere else (i.e your laptop)?

obeleh commented 10 years ago

GCE

justinlindh-wf commented 10 years ago

Same issue. Oddly, this was working about a week ago. Also trying on GCE.

For some reason I don't need to pip install boto to get right to the "No handler was ready to authenticate." exception. Since I've never ran it specifying SECRET_KEY or hardcoding the service account, I didn't follow through on the rest of the ideas above.

Seems like maybe the image is having trouble talking to the metadata server?

Edit: Or not... the original URL containing the "metadata" hostname could be kept in my deployment, instead of hardcoding the IP. See:

root@c3a38eb6fdc7:/docker-registry# ping metadata PING metadata.google.internal (169.254.169.254) 56(84) bytes of data. 64 bytes from metadata.google.internal (169.254.169.254): icmp_req=1 ttl=254 time=0.157 ms 64 bytes from metadata.google.internal (169.254.169.254): icmp_req=2 ttl=254 time=0.209 ms 64 bytes from metadata.google.internal (169.254.169.254): icmp_req=3 ttl=254 time=0.275 ms

proppy commented 10 years ago

This should be fixed with #2

proppy commented 10 years ago

Did a new trusted build of the image: https://index.docker.io/u/google/docker-registry/

Tried:

sudo docker pull google/docker-registry
sudo docker run -e GCS_BUCKET=${DOCKER_BUCKET} -p 5000:5000 google/docker-registry

And it worked for me, can you confirm it fixes your issue?

justinlindh-wf commented 10 years ago

Works perfectly for me now. Great fix, thanks!

obeleh commented 10 years ago

2014-06-06 06:28:06,760 DEBUG: Exception in worker process:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
    __import__(module)
  File "/docker-registry/wsgi.py", line 11, in <module>
    import registry
  File "/docker-registry/registry/__init__.py", line 5, in <module>
    from .tags import *
  File "/docker-registry/registry/tags.py", line 18, in <module>
    store = storage.load()
  File "/docker-registry/lib/storage/__init__.py", line 152, in load
    store = gcs.GSStorage(cfg)
  File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
    BotoStorage.__init__(self, config)
  File "/docker-registry/lib/storage/boto_base.py", line 107, in __init__
    self._config.boto_bucket)
  File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 128, in get_bucket
    bucket.get_all_keys(headers, maxkeys=0)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 471, in get_all_keys
    '', headers, **params)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 399, in _get_all
    query_args=query_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 908, in _mexe
    request.authorize(connection=self)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 379, in authorize
    connection._auth_handler.add_auth(self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_plugin.py", line 45, in add_auth
    self.oauth2_client.GetAuthorizationHeader()
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 344, in GetAuthorizationHeader
    return 'Bearer %s' % self.GetAccessToken().token
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 315, in GetAccessToken
    access_token = self.FetchAccessToken()
  File "/usr/local/lib/python2.7/dist-packages/retry_decorator/retry_decorator.py", line 20, in f_retry
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 520, in FetchAccessToken
    raise GsAccessTokenRefreshError()
TypeError: __init__() takes exactly 2 arguments (1 given)

I have sent a patch for that exception a few days ago. So I guessed metadata issues. And tried pinging:

root@974189e0290c:/docker-registry# ping metadata
ping: unknown host metadata

I suppose it doesn't matter that I'm running docker in a coreos on gce?

obeleh commented 10 years ago

Note that #2 only addresses this issue:

...
File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 197, in __init__
    self.get_credentials(access_key, secret_key, security_token, profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 268, in get_credentials
    if profile_name is None and profile_name_name.upper() in os.environ:
AttributeError: 'NoneType' object has no attribute 'upper'

It doesn't address the metadata or authentication issues that I've described here.

dgageot commented 10 years ago

It doesn't work out of box for me either.

On GCE:

$ docker pull google/docker-registry
...
$ docker run -e GCS_BUCKET=david-quest -p 5000:5000 google/docker-registry

0.1/
computeMetadata/
2014-06-12 15:09:49 [1] [INFO] Starting gunicorn 0.14.5
2014-06-12 15:09:49 [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
2014-06-12 15:09:49 [1] [INFO] Using worker: gevent
2014-06-12 15:09:49 [16] [INFO] Booting worker with pid: 16
2014-06-12 15:09:49,994 DEBUG: path=/david-quest/
2014-06-12 15:09:49,995 DEBUG: auth_path=/david-quest/
2014-06-12 15:09:49,995 DEBUG: path=/david-quest/?max-keys=0
2014-06-12 15:09:49,995 DEBUG: auth_path=/david-quest/?max-keys=0
2014-06-12 15:09:49,995 DEBUG: Method: GET
2014-06-12 15:09:49,995 DEBUG: Path: /david-quest/?max-keys=0
2014-06-12 15:09:49,995 DEBUG: Data:
2014-06-12 15:09:49,995 DEBUG: Headers: {}
2014-06-12 15:09:49,995 DEBUG: Host: storage.googleapis.com
2014-06-12 15:09:49,996 DEBUG: Port: 443
2014-06-12 15:09:49,996 DEBUG: Params: {}
2014-06-12 15:09:49,996 DEBUG: establishing HTTPS connection: host=storage.googleapis.com, kwargs={'port': 443, 'timeout': 70}
2014-06-12 15:09:49,996 DEBUG: Token: None
2014-06-12 15:09:49,996 DEBUG: GetAccessToken: checking cache for key da39a3ee5e6b4b0d3255bfef95601890afd80709
2014-06-12 15:09:49,996 DEBUG: InMemoryTokenCache.GetToken: key=da39a3ee5e6b4b0d3255bfef95601890afd80709 not present
2014-06-12 15:09:49,996 DEBUG: GetAccessToken: token from cache: None
2014-06-12 15:09:49,996 DEBUG: GetAccessToken: fetching fresh access token...
2014-06-12 15:09:50,001 DEBUG: GetAccessToken: fresh access token: None
2014-06-12 15:09:50,001 DEBUG: InMemoryTokenCache.PutToken: key=da39a3ee5e6b4b0d3255bfef95601890afd80709
2014-06-12 15:09:50 [16] [INFO] Worker exiting (pid: 16)
2014-06-12 15:09:50,003 INFO: Worker exiting (pid: 16)
2014-06-12 15:09:51 [1] [INFO] Shutting down: Master
2014-06-12 15:09:51 [1] [INFO] Reason: Worker failed to boot.

dgageot commented 10 years ago

Just realised that da39a3ee5e6b4b0d3255bfef95601890afd80709 is the sha1 of empty string. Maybe it'll help...

proppy commented 10 years ago

@dgageot can you test with the last image version? google/docker-registry:latest, last updated 2014-06-11 23:00:56.

It should gives you more information about what went wrong.

Also make sure you have service account enabled for your GCE instance: https://developers.google.com/compute/docs/authentication#using

obeleh commented 10 years ago

Which ones would be needed? https://www.googleapis.com/auth/devstorage.read_write ?

dgageot commented 10 years ago

It works now!

In the documentation, maybe you should talk about putting a tag on the image before pushing. Something like that:

docker tag dgageot/myimage localhost:5000/dgageot/myimage
docker push localhost:5000/dgageot/myimage

proppy commented 10 years ago

I updated the doc in #4 thanks.

Closing this issue as it seems to work for everybody now.

obeleh commented 10 years ago

Bit late to the show. I've now also tested this and it works. Thanks :)

GoogleCloudPlatform / docker-registry-driver-gcs

docker registry doesn't run out of the box #1