Docker python library broken?

[Cryptophobia]

Hi Deis Devs,

We are encountering an error with the registry-proxy component as it calls the deisbuilder component when we try to pull the example docker application. The reason for the error may be that the Docker python library is picking the wrong api version (v1 vs. v2) when it hits the dockerbuilder with the POST.

INFO [global-inventor]: build global-inventor-d58d621 created
INFO:api.models.app:[global-inventor]: build global-inventor-d58d621 created
INFO [global-inventor]: admin deployed deis/example-go
INFO:api.models.app:[global-inventor]: admin deployed deis/example-go
INFO Pulling Docker image deis/example-go:latest
INFO:registry.dockerclient:Pulling Docker image deis/example-go:latest
INFO Tagging Docker image deis/example-go:latest as localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Tagging Docker image deis/example-go:latest as localhost:5555/global-inventor:v8
INFO Pushing Docker image localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Pushing Docker image localhost:5555/global-inventor:v8
ERROR:backoff:Backing off push(...) for 0.9s (registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused)
INFO Pushing Docker image localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Pushing Docker image localhost:5555/global-inventor:v8
ERROR:backoff:Backing off push(...) for 1.7s (registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused)
INFO Pushing Docker image localhost:5555/global-inventor:v8
INFO:registry.dockerclient:Pushing Docker image localhost:5555/global-inventor:v8
ERROR:backoff:Giving up push(...) after 3 tries (registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused)
INFO [global-inventor]: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
INFO:api.models.app:[global-inventor]: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
ERROR:root:Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
Traceback (most recent call last):
  File "/app/api/models/release.py", line 89, in new
    release.publish()
  File "/app/api/models/release.py", line 136, in publish
    publish_release(source_image, self.image, deis_registry, self.get_registry_auth())
  File "/app/registry/dockerclient.py", line 195, in publish_release
    return DockerClient().publish_release(source, target, deis_registry, creds)
  File "/app/registry/dockerclient.py", line 114, in publish_release
    self.push("{}/{}".format(self.registry, name), tag)
  File "/usr/local/lib/python3.5/dist-packages/backoff/_sync.py", line 85, in retry
    ret = target(*args, **kwargs)
  File "/app/registry/dockerclient.py", line 131, in push
    log_output(stream, 'push', repo, tag)
  File "/app/registry/dockerclient.py", line 174, in log_output
    stream_error(chunk, operation, repo, tag)
  File "/app/registry/dockerclient.py", line 191, in stream_error
    raise RegistryException(message)
registry.dockerclient.RegistryException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/models/build.py", line 63, in create
    source_version=self.version
  File "/app/api/models/release.py", line 96, in new
    raise DeisException(str(e)) from e
api.exceptions.DeisException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/app/api/views.py", line 185, in create
    return super(AppResourceViewSet, self).create(request, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/app/api/viewsets.py", line 21, in perform_create
    self.post_save(obj)
  File "/app/api/views.py", line 268, in post_save
    self.release = build.create(self.request.user)
  File "/app/api/models/build.py", line 79, in create
    raise DeisException(str(e)) from e
api.exceptions.DeisException: Put http://localhost:5555/v1/repositories/global-inventor/: dial tcp [::1]:5555: getsockopt: connection refused
10.42.0.8 "POST /v2/apps/global-inventor/builds/ HTTP/1.1" 400 124 "Deis Client v2.13.0"

Notice the 10.42.0.8 "POST /v2/apps/global-inventor/builds/ HTTP/1.1" 400 124 "Deis Client v2.13.0" at the end of the log.

Is this related to https://github.com/deis/registry/issues/64#issuecomment-249275607 and how do we get around it?

[bacongobbler]

Yes, this is indeed a duplicate of deis/registry#64. Did you try out the workarounds posted further below the comment chain?

[Cryptophobia]

@bacongobbler Hi and thanks for the help!

Actually we tried the workaround that you posted with the patch that is applied on top of workflow-dev chart posted here: https://github.com/deis/registry/issues/64#issuecomment-255222613

However, this did not work for us as we were getting the same error with the docker client using the wrong version of the docker registry api.

To fix this we went back to using the --insecure-registry flags on our docker startup scripts and reverted to using the older chart for workflow-v2.7.0 with the patch and instructions that you provided in https://github.com/deis/registry/issues/64#issuecomment-254916941

What would be another way we can fix this issue because we want to use the latest chart workflow-2.9.0 if possible.

Would the way to go be to apply the same patch on top of workflow-v2.9.0 and use the --insecure-registry flags?

[bacongobbler]

Another way to fix this would be to use an external container registry like ECR or GCR. That way you bypass the registry-proxy entirely which depends on host port binding to work.

[Cryptophobia]

@bacongobbler 🥇 We deployed with a external ECR like you said and we managed to bypass the registry-proxy completely. No need for the workarounds anymore. Thanks!