cannot upload docker container to registry

[DavidSie]

When I build an up with buildpack it works, but when I want to build container I cannot upload it to the registry

 kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4+coreos.0", GitCommit:"be9bf3e842a90537e48361aded2872e389e902e7", GitTreeState:"clean", BuildDate:"2016-08-02T00:54:53Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

deis version
v2.4.0

     git push deis master
Counting objects: 589, done.
Compressing objects: 100% (416/416), done.
Writing objects: 100% (589/589), 2.47 MiB, done.
Total 589 (delta 46), reused 581 (delta 42)
Starting build... but first, coffee!
Step 1 : FROM ruby:2.0.0-p576
---> a137b6df82e8
Step 2 : COPY . /app
---> Using cache
---> a7107ea0f79a
Step 3 : WORKDIR /app
---> Using cache
---> ba2d0c3222ec
Step 4 : EXPOSE 3000
---> Using cache
---> 18f7fb188ed3
Step 5 : CMD while true; do echo hello world; sleep 1; done
---> Using cache
---> 4e22b0487484
Successfully built 4e22b0487484
Pushing to registry
{"errorDetail":{"message":"Put http://localhost:5555/v1/repositories/spree/: dial tcp 127.0.0.1:5555: getsockopt: connection refused"},"error":"Put http://localhost:5555/v1/repositories/spree/: dial tcp 127.0.0.remote: getsockopt: connection refused"}

I know that there are environmental variables to point this address:

     Environment Variables:
      DEIS_REGISTRY_SERVICE_HOST:   localhost
      DEIS_REGISTRY_SERVICE_PORT:   5555

but I don't understand why, since none of the pods, and none of the services is listening on 5555

services

kubectl get services --namespace=deis
NAME                     CLUSTER-IP   EXTERNAL-IP   PORT(S)                            AGE
deis-builder             10.3.0.233   <none>        2222/TCP                           1d
deis-controller          10.3.0.23    <none>        80/TCP                             1d
deis-database            10.3.0.253   <none>        5432/TCP                           1d
deis-logger              10.3.0.221   <none>        80/TCP                             1d
deis-logger-redis        10.3.0.148   <none>        6379/TCP                           1d
deis-minio               10.3.0.232   <none>        9000/TCP                           1d
deis-monitor-grafana     10.3.0.113   <none>        80/TCP                             1d
deis-monitor-influxapi   10.3.0.234   <none>        80/TCP                             1d
deis-monitor-influxui    10.3.0.141   <none>        80/TCP                             1d
deis-nsqd                10.3.0.82    <none>        4151/TCP,4150/TCP                  1d
deis-registry            10.3.0.188   <none>        80/TCP                             1d
deis-router              10.3.0.133   <pending>     80/TCP,443/TCP,2222/TCP,9090/TCP   1d
deis-workflow-manager    10.3.0.34    <none>        80/TCP                             1d

pods

kubectl describe  pods deis-registry-3758253254-3gtjo   --namespace=deis 
Name:       deis-registry-3758253254-3gtjo
Namespace:  deis
Node:       10.63.11.75/10.63.11.75
Start Time: Mon, 22 Aug 2016 10:36:12 +0000
Labels:     app=deis-registry
        pod-template-hash=3758253254
Status:     Running
IP:     10.2.12.12
Controllers:    ReplicaSet/deis-registry-3758253254
Containers:
  deis-registry:
    Container ID:   docker://78d6d569eefac3766e4b921f21b7847d36866a266ae76424d7d6e572bb2f5979
    Image:      quay.io/deis/registry:v2.2.0
    Image ID:       docker://sha256:0eb83b180d1aa993fcdd715e4b919b4867051d4f35a813a56eec04ae0705d3d1
    Port:       5000/TCP
    State:      Running
      Started:      Mon, 22 Aug 2016 10:43:05 +0000
    Ready:      True
    Restart Count:  0
    Liveness:       http-get http://:5000/v2/ delay=1s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:5000/v2/ delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
      REGISTRY_STORAGE_DELETE_ENABLED:  true
      REGISTRY_LOG_LEVEL:       info
      REGISTRY_STORAGE:         minio
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  registry-storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  registry-creds:
    Type:   Secret (a volume populated by a Secret)
    SecretName: objectstorage-keyfile
  deis-registry-token-inyyj:
    Type:   Secret (a volume populated by a Secret)
    SecretName: deis-registry-token-inyyj
QoS Tier:   BestEffort
No events.

kubectl describe  pods deis-registry-proxy-cpu68    --namespace=deis 
Name:       deis-registry-proxy-cpu68
Namespace:  deis
Node:       10.63.11.76/10.63.11.76
Start Time: Mon, 22 Aug 2016 10:36:31 +0000
Labels:     app=deis-registry-proxy
        heritage=deis
Status:     Running
IP:     10.2.63.4
Controllers:    DaemonSet/deis-registry-proxy
Containers:
  deis-registry-proxy:
    Container ID:   docker://dc29ab400a06ae5dc1407c7f1fb0880d4257720170eded6a7f8cde5431fa9570
    Image:      quay.io/deis/registry-proxy:v1.0.0
    Image ID:       docker://sha256:fde297ec95aa244e5be48f438de39a13dae16a1593b3792d8c10cd1d7011f8d1
    Port:       80/TCP
    Limits:
      cpu:  100m
      memory:   50Mi
    Requests:
      cpu:      100m
      memory:       50Mi
    State:      Running
      Started:      Mon, 22 Aug 2016 10:38:32 +0000
    Ready:      True
    Restart Count:  0
    Environment Variables:
      REGISTRY_HOST:    $(DEIS_REGISTRY_SERVICE_HOST)
      REGISTRY_PORT:    $(DEIS_REGISTRY_SERVICE_PORT)
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  default-token-tk993:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-tk993
QoS Tier:   Guaranteed
No events.

[bacongobbler]

You can fetch the chart locally via helm fetch deis/workflow --version=v2.9.1 --untar, modify the chart with the patch (which you'll have to manually apply since it's not in git), then install it :)

[ineu]

Thank you

[anubhavmishra]

Patched with @bacongobbler's suggested fixes https://github.com/deis/registry/issues/64#issuecomment-258003967 latest helm workflow charts: https://github.com/anubhavmishra/workflow. Also make sure you are using insecure registry option for Docker suggested here: https://deis.com/docs/workflow/en/v2.2.0/installing-workflow/system-requirements/#docker-insecure-registry

[IlyaSemenov]

For v2.15.0, the recipe will be:

helm fetch deis/workflow --version=v2.15.0 --untar
cd workflow
curl https://gist.githubusercontent.com/IlyaSemenov/a8f467934cb5f1f0963469cd3eb32ace/raw/b3e8fcb5dd9094b50014177f5db72210b2949883/0001-Remove-proxy.patch|patch -p1
helm upgrade deis .

Don't forget to enable insecure registry /lib/systemd/system/docker.service at your Docker host(s):

ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=10.43.0.0/16

[felixbuenemann]

Removing the registry proxy should no longer be needed with current versions of the deis helm charts, you can set the following in your deis-workflow values.yml if you are using CNI:

global:
  host_port: 5555
  use_cni: true
  registry_proxy_bind_addr: "127.0.0.1:5555"

[IlyaSemenov]

It's not working on Kubernetes 1.5.4 provisioned with Rancher 1.6.2 (latest).

I think this is the related issue https://github.com/rancher/rancher/issues/5857.

[ghost]

I'm new to Deis and I'm encountering all kind of problems in my journey to deploy Deis in AWS. The last one is when I tried to deploy a Doker image to Deis. For example for a pgadmin4 Docker image when running deis pull ephillipe/pgadmin4, I'm getting this error: Creating build... Error: Unknown Error (400): {"detail":"Put http://127.0.0.1:5555/v1/repositories/pgadmin4/: dial tcp 127.0.0.1:5555: getsockopt: connection refused"} I checked the running daemonsets: kubectl --namespace=deis get daemonsets and I'm getting:

deis-logger-fluentd     2         2         2         2            2           <none>          6d
deis-monitor-telegraf   2         2         2         2            2           <none>          6d
deis-registry-proxy     0         0         0         0            0           <none>          6d

So clearly the problem is because deis-registry-proxy is not running.

Can anyone help me with this issue? How can I start deis-registry-proxy or if that's not the solution how can I deploy a docker image then?

[felixbuenemann]

@IulianParaian I would try the deis slack for troubleshooting. Might be your registry proxies are crashing because the internal registry is unreachable.

[ghost]

@felixbuenemann I did tried the said slack first but didn't get any responses. And I also couldn't find a good documentation or a simple example on how to deploy an app from Docker image/ Docker file. I'm not referring to the official Deis documentation because there are just 3 lines of text with one command line that should work, but obviously it is not. So maybe some more detailed tutorials with some possible troubleshooting would help.

PS: I raised another issue on Workflow repo regarding an installation using off cluster storage, but no response there either. And for that I also followed the official steps.

[bacongobbler]

And I also couldn't find a good documentation or a simple example on how to deploy an app from Docker image/ Docker file.

I understand your frustration, though if the documentation is lacking, there are example applications provided for nearly any configuration you're looking for in the github org, and we do link to those example applications in the documentation. For example: https://github.com/deis/example-dockerfile-http

Have you taken a look at the troubleshooting documentation? That should help give you a general guideline on how you can self-troubleshoot why your cluster is not working the way it should. If all else fails you can troubleshoot directly using kubectl following kubernetes' documentation.

[ghost]

Hi @bacongobbler, thank you for the answer. I did troubleshooting my kubernetes and noticed that deis-registry-proxy component was not running. This example https://github.com/deis/example-dockerfile-http is one that I tried.

As I am writing this I went to check the deis pods again and surprisingly, I have 2 deis-registry-proxy instances running. That is strange, I didn't change anything since I posted the issue. Thanks again.