TypeError: __init__() got an unexpected keyword argument 'server_hostname'

[carletes]

I'm running docker-registry in mirror mode under Ubuntu 14.10. When I do a docker pull <image name>, I see the following error in the logs of docker-registry, and then the Docker daemon fetches the image from the Internet instead:

[2015-03-04 09:16:05 +0000] [4801] [INFO] Starting gunicorn 19.1.1
[2015-03-04 09:16:05 +0000] [4801] [INFO] Listening at: http://0.0.0.0:5000 (4801)
[2015-03-04 09:16:05 +0000] [4801] [INFO] Using worker: gevent
[2015-03-04 09:16:05 +0000] [4806] [INFO] Booting worker with pid: 4806
[2015-03-04 09:16:05 +0000] [4807] [INFO] Booting worker with pid: 4807
[2015-03-04 09:16:05 +0000] [4808] [INFO] Booting worker with pid: 4808
[2015-03-04 09:16:05 +0000] [4809] [INFO] Booting worker with pid: 4809
04/Mar/2015:09:16:06 +0000 WARNING: Cache storage disabled!
04/Mar/2015:09:16:06 +0000 WARNING: LRU cache disabled!
04/Mar/2015:09:16:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
04/Mar/2015:09:16:06 +0000 WARNING: Cache storage disabled!
04/Mar/2015:09:16:06 +0000 WARNING: LRU cache disabled!
04/Mar/2015:09:16:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
04/Mar/2015:09:16:06 +0000 WARNING: Cache storage disabled!
04/Mar/2015:09:16:06 +0000 WARNING: LRU cache disabled!
04/Mar/2015:09:16:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
04/Mar/2015:09:16:06 +0000 WARNING: Cache storage disabled!
04/Mar/2015:09:16:06 +0000 WARNING: LRU cache disabled!
04/Mar/2015:09:16:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
04/Mar/2015:09:17:54 +0000 DEBUG: args = {'image_id': u'c90d655b99b2ec5b7e94d38c87f92dce015c17a313caeaae0e980d9b9bed8444'}
04/Mar/2015:09:17:54 +0000 DEBUG: Auth Token = Token signature=9802a55c955ec27cd1a2655e1fe98372bd5f616a,repository="library/debian",access=read
04/Mar/2015:09:17:54 +0000 DEBUG: auth = {u'access': u'read', u'repository': u'library/debian', u'signature': u'9802a55c955ec27cd1a2655e1fe98372bd5f616a'}
04/Mar/2015:09:17:54 +0000 INFO: Starting new HTTPS connection (1): index.docker.io
2015-03-04 09:17:55,040 ERROR: Exception on /v1/images/c90d655b99b2ec5b7e94d38c87f92dce015c17a313caeaae0e980d9b9bed8444/ancestry [GET]
Traceback (most recent call last):
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/docker_registry/toolkit.py", line 279, in wrapper
    if check_signature() is True or check_token(kwargs) is True:
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/docker_registry/toolkit.py", line 231, in check_token
    if validate_token(auth) is False:
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/docker_registry/toolkit.py", line 159, in validate_token
    resp = requests.get(url, verify=True, headers=headers)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 190, in connect
    ssl_version=resolved_ssl_version)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 124, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
TypeError: __init__() got an unexpected keyword argument 'server_hostname'
04/Mar/2015:09:17:55 +0000 ERROR: Exception on /v1/images/c90d655b99b2ec5b7e94d38c87f92dce015c17a313caeaae0e980d9b9bed8444/ancestry [GET]
Traceback (most recent call last):
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/docker_registry/toolkit.py", line 279, in wrapper
    if check_signature() is True or check_token(kwargs) is True:
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/docker_registry/toolkit.py", line 231, in check_token
    if validate_token(auth) is False:
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/docker_registry/toolkit.py", line 159, in validate_token
    resp = requests.get(url, verify=True, headers=headers)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 190, in connect
    ssl_version=resolved_ssl_version)
  File "/srv/docker-registry/virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 124, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
TypeError: __init__() got an unexpected keyword argument 'server_hostname'
127.0.0.1 - - [04/Mar/2015:09:17:55 +0000] "GET /v1/images/c90d655b99b2ec5b7e94d38c87f92dce015c17a313caeaae0e980d9b9bed8444/ancestry HTTP/1.1" 500 291 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.16.0-23-generic os/linux arch/amd64"

I'm running Ubuntu 14.10 64-bit, and using the official Docker repositories. The output of docker version is the following:

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

I have set up the Docker daemon to use my docker-registry instance by using this /etc/default/docker:

# Docker Upstart and SysVinit configuration file

# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"

# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

DOCKER_OPTS="--registry-mirror=http://localhost:5000"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"

The docker-registry instance is started like this (using daemontools):

#!/bin/sh

exec 2>&1

exec \
  sudo -u registry \
    env \
      DOCKER_REGISTRY_CONFIG=/srv/docker-registry/config.yaml \
      PATH="/srv/docker-registry/virtualenv/bin" \
    docker-registry \
      "$@"

The configuration file I'm using for docker-registry is the following:

# The `common' part is automatically included (and possibly overriden by all
# other flavors)
common: &common
    index_endpoint: https://index.docker.io
    mirroring:
        source: https://registry-1.docker.io
        source_index: https://index.docker.io
        tags_cache_ttl: 17280000  # 200 days

dev:
    <<: *common
    storage: local
    storage_path: /srv/docker-registry/data
    loglevel: debug

The Python virtualenv docker-registry runs under is the following (this is the output of pip freeze):

Flask==0.10.1
Jinja2==2.7.3
M2Crypto==0.22.3
MarkupSafe==0.23
PyYAML==3.11
SQLAlchemy==0.9.4
Werkzeug==0.10.1
argparse==1.2.1
backports.lzma==0.0.3
blinker==1.3
boto==2.34.0
docker-registry==0.9.1
docker-registry-core==2.0.3
gevent==1.0.1
greenlet==0.4.5
gunicorn==19.1.1
itsdangerous==0.24
redis==2.10.3
requests==2.3.0
simplejson==3.6.2
wsgiref==0.1.2

OpenSSL comes from the Ubuntu installation (don't know whether this is relevant), and its version number is:

OpenSSL 1.0.1f 6 Jan 2014

Interestingly, I'm running the same setup under Ubuntu 14.04 LTS on another system, and things work perfectly well there.

Any hints?

[noxiouz]

Seems it's connected with a versions of python in 14.04 & 14.10. ssl package was changed between them. Python requests of your version rely on the fact that ssl package from standart library hasn't got SSLContext: https://github.com/kennethreitz/requests/blob/v2.3.0/requests/packages/urllib3/util/ssl_.py#L95 But it was backported from python 3.2+ to python 2.7.9. So the bottom line is you can try to update python requests :-)

[carletes]

Thanks, @noxiouz! I tried running it with several versions of requests, but to no avail :( I'm running it now in a container, anyway, so problem bypassed.

[benley]

For anybody else who runs into this, it appears that running gunicorn with eventlet workers instead of gevent may be a viable workaround.

e.g.

gunicorn --access-logfile - --error-logfile - -k eventlet -b 0.0.0.0:5000 \
         -w 4 --max-requests 100 docker_registry.wsgi:application

[akaspin]

Why it's closed? Problem persists.

[dmp42]

@akaspin the OP closed it himself.

The recommended way to run the registry is from a container.

This type of situation is exactly why running directly on your system is not recommended.

Nevertheless:

• this probably affect only when ran as a mirror
• if you really want to run this on your system, try other (more recent) versions of requests
• I'm open to merging a PR that would upgrade requests, but it will probably require some extra work to make sure it does not cause regressions. There is an open issue here to do that: #611

[benley]

This bug also manifests itself if you're using an S3 storage backend, for what it's worth. Probably also any other boto-driven backend where SSL is involved. My solution was, as you recommended, to run it from the docker image.