Closed obeleh closed 10 years ago
So I've enabled logging in gunicorn (in run.sh):
exec gunicorn --log-level debug --access-logfile - --debug --max-requests 100 --graceful-timeout $GUNICORN_GRACEFUL_TIMEOUT -t $GUNICORN_SILENT_TIMEOUT -k gevent -b 0.0.0.0:$REGISTRY_PORT -w $GUNICORN_WORKERS wsgi:application
And that results in this output:
2014-06-03 07:19:02,439 DEBUG: Exception in worker process:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
super(GeventWorker, self).init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
__import__(module)
File "/docker-registry/wsgi.py", line 11, in <module>
import registry
File "/docker-registry/registry/__init__.py", line 5, in <module>
from .tags import *
File "/docker-registry/registry/tags.py", line 18, in <module>
store = storage.load()
File "/docker-registry/lib/storage/__init__.py", line 152, in load
store = gcs.GSStorage(cfg)
File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
BotoStorage.__init__(self, config)
File "/docker-registry/lib/storage/boto_base.py", line 105, in __init__
self._boto_conn = self.makeConnection()
File "/docker-registry/lib/storage/gcs.py", line 34, in makeConnection
return uri.connect()
File "/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 140, in connect
**connection_args)
File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 47, in __init__
suppress_consec_slashes=suppress_consec_slashes)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 190, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 558, in __init__
profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 197, in __init__
self.get_credentials(access_key, secret_key, security_token, profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 268, in get_credentials
if profile_name is None and profile_name_name.upper() in os.environ:
AttributeError: 'NoneType' object has no attribute 'upper'
Ah I remember. Boto issue: https://github.com/boto/boto/pull/2302
pip install -U boto
fixes this. Probably the docker image needs this as well.
Which brings me to the next problem:
2014-06-03 07:29:07,878 DEBUG: Exception in worker process:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
super(GeventWorker, self).init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
__import__(module)
File "/docker-registry/wsgi.py", line 11, in <module>
import registry
File "/docker-registry/registry/__init__.py", line 5, in <module>
from .tags import *
File "/docker-registry/registry/tags.py", line 18, in <module>
store = storage.load()
File "/docker-registry/lib/storage/__init__.py", line 152, in load
store = gcs.GSStorage(cfg)
File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
BotoStorage.__init__(self, config)
File "/docker-registry/lib/storage/boto_base.py", line 105, in __init__
self._boto_conn = self.makeConnection()
File "/docker-registry/lib/storage/gcs.py", line 34, in makeConnection
return uri.connect()
File "/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 140, in connect
**connection_args)
File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 47, in __init__
suppress_consec_slashes=suppress_consec_slashes)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 190, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 572, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/local/lib/python2.7/dist-packages/boto/auth.py", line 883, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 3 handlers were checked. ['HmacAuthV1Handler', 'OAuth2Auth', 'OAuth2ServiceAccountAuth'] Check your credentials
As a quick patch I fix the oath2 plugin since we're going to need to connect to the metadata server anyway.
vim /usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py
And change this:
META_TOKEN_URI = ('http://metadata/computeMetadata/v1/instance/'
'service-accounts/default/token')
To this:
META_TOKEN_URI = ('http://169.254.169.254/computeMetadata/v1/instance/'
'service-accounts/default/token')
So now the boto library is skipping the google provider because it requires a service account.
Quick hack to make it work. I have no idea how and where I should add this. Editing config.yml
didn't work.
File: /usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_plugin.py
if (provider.name == 'google'):
config.add_section('GoogleCompute')
config.set('GoogleCompute', 'service_account', 'XXXXXX@developer.gserviceaccount.com')
Apparently I forgot to save /usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py
and I got this message:
2014-06-03 08:39:14,995 DEBUG: Exception in worker process:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
super(GeventWorker, self).init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
__import__(module)
File "/docker-registry/wsgi.py", line 11, in <module>
import registry
File "/docker-registry/registry/__init__.py", line 5, in <module>
from .tags import *
File "/docker-registry/registry/tags.py", line 18, in <module>
store = storage.load()
File "/docker-registry/lib/storage/__init__.py", line 152, in load
store = gcs.GSStorage(cfg)
File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
BotoStorage.__init__(self, config)
File "/docker-registry/lib/storage/boto_base.py", line 107, in __init__
self._config.boto_bucket)
File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 128, in get_bucket
bucket.get_all_keys(headers, maxkeys=0)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 471, in get_all_keys
'', headers, **params)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 399, in _get_all
query_args=query_args)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 908, in _mexe
request.authorize(connection=self)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 379, in authorize
connection._auth_handler.add_auth(self, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_plugin.py", line 46, in add_auth
self.oauth2_client.GetAuthorizationHeader()
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 344, in GetAuthorizationHeader
return 'Bearer %s' % self.GetAccessToken().token
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 315, in GetAccessToken
access_token = self.FetchAccessToken()
File "/usr/local/lib/python2.7/dist-packages/retry_decorator/retry_decorator.py", line 20, in f_retry
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 520, in FetchAccessToken
raise GsAccessTokenRefreshError()
TypeError: __init__() takes exactly 2 arguments (1 given)
So change this (/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py):
except Exception:
raise GsAccessTokenRefreshError()
into this:
except Exception as ex:
raise GsAccessTokenRefreshError(str(ex))
And then got the correct error message:
GsAccessTokenRefreshError: Unable to find the server at metadata
After setting the metadata server to an IP adress it worked.
Up to now I've hardcoded my GCS_BUCKET and SECRET_KEY into config.yml
. Time to set it back to original and try to run:
As described on the main page but with two changes in order to make it work.
docker run -i -t -p 10.240.168.100:5000:5000 -e SECRET_KEY=XXXXXX -e GCS_BUCKET=osdocker google/docker-registry
So I suppose adding secret_key and hardcoding my service account shouldn't be done. I suppose this should have been discovered in some manner. I don't know how. All I know is, without these settings it won't work.
Are you trying to get the registry running on GCE or somewhere else (i.e your laptop)?
GCE
Same issue. Oddly, this was working about a week ago. Also trying on GCE.
For some reason I don't need to pip install boto to get right to the "No handler was ready to authenticate." exception. Since I've never ran it specifying SECRET_KEY or hardcoding the service account, I didn't follow through on the rest of the ideas above.
Seems like maybe the image is having trouble talking to the metadata server?
Edit: Or not... the original URL containing the "metadata" hostname could be kept in my deployment, instead of hardcoding the IP. See:
root@c3a38eb6fdc7:/docker-registry# ping metadata PING metadata.google.internal (169.254.169.254) 56(84) bytes of data. 64 bytes from metadata.google.internal (169.254.169.254): icmp_req=1 ttl=254 time=0.157 ms 64 bytes from metadata.google.internal (169.254.169.254): icmp_req=2 ttl=254 time=0.209 ms 64 bytes from metadata.google.internal (169.254.169.254): icmp_req=3 ttl=254 time=0.275 ms
This should be fixed with #2
Did a new trusted build of the image: https://index.docker.io/u/google/docker-registry/
Tried:
sudo docker pull google/docker-registry
sudo docker run -e GCS_BUCKET=${DOCKER_BUCKET} -p 5000:5000 google/docker-registry
And it worked for me, can you confirm it fixes your issue?
Works perfectly for me now. Great fix, thanks!
2014-06-06 06:28:06,760 DEBUG: Exception in worker process:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 98, in init_process
super(GeventWorker, self).init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 99, in init_process
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 101, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 24, in load
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 291, in import_app
__import__(module)
File "/docker-registry/wsgi.py", line 11, in <module>
import registry
File "/docker-registry/registry/__init__.py", line 5, in <module>
from .tags import *
File "/docker-registry/registry/tags.py", line 18, in <module>
store = storage.load()
File "/docker-registry/lib/storage/__init__.py", line 152, in load
store = gcs.GSStorage(cfg)
File "/docker-registry/lib/storage/gcs.py", line 21, in __init__
BotoStorage.__init__(self, config)
File "/docker-registry/lib/storage/boto_base.py", line 107, in __init__
self._config.boto_bucket)
File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 128, in get_bucket
bucket.get_all_keys(headers, maxkeys=0)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 471, in get_all_keys
'', headers, **params)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 399, in _get_all
query_args=query_args)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 908, in _mexe
request.authorize(connection=self)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 379, in authorize
connection._auth_handler.add_auth(self, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_plugin.py", line 45, in add_auth
self.oauth2_client.GetAuthorizationHeader()
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 344, in GetAuthorizationHeader
return 'Bearer %s' % self.GetAccessToken().token
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 315, in GetAccessToken
access_token = self.FetchAccessToken()
File "/usr/local/lib/python2.7/dist-packages/retry_decorator/retry_decorator.py", line 20, in f_retry
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oauth2_plugin/oauth2_client.py", line 520, in FetchAccessToken
raise GsAccessTokenRefreshError()
TypeError: __init__() takes exactly 2 arguments (1 given)
I have sent a patch for that exception a few days ago. So I guessed metadata issues. And tried pinging:
root@974189e0290c:/docker-registry# ping metadata
ping: unknown host metadata
I suppose it doesn't matter that I'm running docker in a coreos on gce?
Note that #2 only addresses this issue:
...
File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 197, in __init__
self.get_credentials(access_key, secret_key, security_token, profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 268, in get_credentials
if profile_name is None and profile_name_name.upper() in os.environ:
AttributeError: 'NoneType' object has no attribute 'upper'
It doesn't address the metadata or authentication issues that I've described here.
It doesn't work out of box for me either.
On GCE:
$ docker pull google/docker-registry
...
$ docker run -e GCS_BUCKET=david-quest -p 5000:5000 google/docker-registry
0.1/
computeMetadata/
2014-06-12 15:09:49 [1] [INFO] Starting gunicorn 0.14.5
2014-06-12 15:09:49 [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
2014-06-12 15:09:49 [1] [INFO] Using worker: gevent
2014-06-12 15:09:49 [16] [INFO] Booting worker with pid: 16
2014-06-12 15:09:49,994 DEBUG: path=/david-quest/
2014-06-12 15:09:49,995 DEBUG: auth_path=/david-quest/
2014-06-12 15:09:49,995 DEBUG: path=/david-quest/?max-keys=0
2014-06-12 15:09:49,995 DEBUG: auth_path=/david-quest/?max-keys=0
2014-06-12 15:09:49,995 DEBUG: Method: GET
2014-06-12 15:09:49,995 DEBUG: Path: /david-quest/?max-keys=0
2014-06-12 15:09:49,995 DEBUG: Data:
2014-06-12 15:09:49,995 DEBUG: Headers: {}
2014-06-12 15:09:49,995 DEBUG: Host: storage.googleapis.com
2014-06-12 15:09:49,996 DEBUG: Port: 443
2014-06-12 15:09:49,996 DEBUG: Params: {}
2014-06-12 15:09:49,996 DEBUG: establishing HTTPS connection: host=storage.googleapis.com, kwargs={'port': 443, 'timeout': 70}
2014-06-12 15:09:49,996 DEBUG: Token: None
2014-06-12 15:09:49,996 DEBUG: GetAccessToken: checking cache for key da39a3ee5e6b4b0d3255bfef95601890afd80709
2014-06-12 15:09:49,996 DEBUG: InMemoryTokenCache.GetToken: key=da39a3ee5e6b4b0d3255bfef95601890afd80709 not present
2014-06-12 15:09:49,996 DEBUG: GetAccessToken: token from cache: None
2014-06-12 15:09:49,996 DEBUG: GetAccessToken: fetching fresh access token...
2014-06-12 15:09:50,001 DEBUG: GetAccessToken: fresh access token: None
2014-06-12 15:09:50,001 DEBUG: InMemoryTokenCache.PutToken: key=da39a3ee5e6b4b0d3255bfef95601890afd80709
2014-06-12 15:09:50 [16] [INFO] Worker exiting (pid: 16)
2014-06-12 15:09:50,003 INFO: Worker exiting (pid: 16)
2014-06-12 15:09:51 [1] [INFO] Shutting down: Master
2014-06-12 15:09:51 [1] [INFO] Reason: Worker failed to boot.
Just realised that da39a3ee5e6b4b0d3255bfef95601890afd80709
is the sha1 of empty string. Maybe it'll help...
@dgageot can you test with the last image version?
google/docker-registry:latest
, last updated 2014-06-11 23:00:56.
It should gives you more information about what went wrong.
Also make sure you have service account enabled for your GCE instance: https://developers.google.com/compute/docs/authentication#using
Which ones would be needed? https://www.googleapis.com/auth/devstorage.read_write ?
It works now!
In the documentation, maybe you should talk about putting a tag on the image before pushing. Something like that:
docker tag dgageot/myimage localhost:5000/dgageot/myimage
docker push localhost:5000/dgageot/myimage
I updated the doc in #4 thanks.
Closing this issue as it seems to work for everybody now.
Bit late to the show. I've now also tested this and it works. Thanks :)
I'm going to add more info as I'm working through the steps needed to get it to run