search

deis / registry

Docker registry for Deis Workflow.
https://deis.com
MIT License
16 stars 24 forks source link

Can't startup registry with Azure backend #56

Closed monaka closed 8 years ago

monaka commented 8 years ago

I'm trying to install Deis with Azure Blob but failed.

I re-generated from chart, uninstalled, installed after editing ~/.helmc/workspace/charts/workflow-v2.0.0/tpl/generate_params.toml

The log is like this:

 $ kubectl logs -f deis-registry-34xc6 --namespace=deis 
2016/06/20 00:42:05 INFO: Starting registry...
2016/06/20 00:42:05 INFO: using azure as the backend

But it can't be started up....

$ kubectl get pods --namespace=deis
(snip)
deis-registry-34xc6           0/1       CrashLoopBackOff   11         20m

For checking my account information, I run the docker registry on my node manually with my info. It seems this works well.

$ docker run -it --rm -p 5000:5000      -e REGISTRY_STORAGE=azure      -e REGISTRY_STORAGE_AZURE_ACCOUNTNAME="{account_name_censored}"      -e 
REGISTRY_STORAGE_AZURE_ACCOUNTKEY="{account_key_censored}"      -e REGISTRY_STORAGE
_AZURE_CONTAINER="registry"      --name=registry      registry:2
WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.  go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1                                                                                      
INFO[0000] redis not configured                          go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1
INFO[0000] Starting upload purge in 23m0s                go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1
INFO[0000] listening on [::]:5000                        go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1

I think there have some troubles in /bin/registry. But I can't determine as there have few logs. Could you tell me some tips for shooting this kind of trouble?

kmala commented 8 years ago

i had just tried using azure on workflow-v2.0.0 chart and it seems to be working....can you say what kubectl describe pod <registry-pod> --namespace=deis say and also can you get the logs of the container by deleting the pod and before it restarting/going into crashloop because sometime kubectl misses some logs when the container is in crashloopbackoff.

monaka commented 8 years ago

The result of describe pods like this. Seems registry daemon didn't spawn...

$ kubectl describe pods deis-registry-e3ztm  --namespace deis
Name:           deis-registry-e3ztm
Namespace:      deis
Node:           10.0.2.4/10.0.2.4
Start Time:     Mon, 20 Jun 2016 03:03:02 +0000
Labels:         app=deis-registry
Status:         Running
IP:             172.16.43.10
Controllers:    ReplicationController/deis-registry
Containers:
  deis-registry:
    Container ID:       docker://960a9c38f1d365667ebc5061a7106a73286fe28c4efdcf651feef2ef40c82542
    Image:              quay.io/deis/registry:v2.0.0
    Image ID:           docker://sha256:4d0dfeacd8e622bee7446ff905b9c76ff5c2c94bfaaceef094e0a52296b0d098
    Port:               5000/TCP
    QoS Tier:
      memory:           BestEffort
      cpu:              BestEffort
    State:              Waiting
      Reason:           CrashLoopBackOff
    Last State:         Terminated
      Reason:           Error
      Exit Code:        2
      Started:          Mon, 20 Jun 2016 03:04:12 +0000
      Finished:         Mon, 20 Jun 2016 03:04:22 +0000
    Ready:              False
    Restart Count:      3
    Liveness:           http-get http://:5000/v2/ delay=1s timeout=1s period=10s #success=1 #failure=3
    Readiness:          http-get http://:5000/v2/ delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
      REGISTRY_STORAGE_DELETE_ENABLED:  true
      REGISTRY_LOG_LEVEL:               info
      REGISTRY_STORAGE:                 azure
Conditions:
  Type          Status
  Ready         False
Volumes:
  registry-storage:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  registry-creds:
    Type:       Secret (a volume populated by a Secret)
    SecretName: objectstorage-keyfile
  deis-registry-token-p6m3l:
    Type:       Secret (a volume populated by a Secret)
    SecretName: deis-registry-token-p6m3l
Events:
  FirstSeen     LastSeen        Count   From                    SubobjectPath                   Type            Reason          Message
  ---------     --------        -----   ----                    -------------                   --------        ------          -------
  1m            1m              1       {default-scheduler }                                    Normal          Scheduled       Successfully assigned deis-registry-e3ztm to 10.0.2.4
  1m            1m              1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Created         Created container with docker id e6b44596f083
  1m            1m              1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Started         Started container with docker id e6b44596f083
  32s           32s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Killing         Killing container with docker id e6b44596f083: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
  32s           32s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Started         Started container with docker id 23e5fde34cb5
  32s           32s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Created         Created container with docker id 23e5fde34cb5
  22s           22s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Killing         Killing container with docker id 23e5fde34cb5: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
  22s           22s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Created         Created container with docker id 1bd56ff56fa2
  22s           22s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Started         Started container with docker id 1bd56ff56fa2
  1m            12s             4       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Pulled          Container image "quay.io/deis/registry:v2.0.0" already present on machine
  12s           12s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Killing         Killing container with docker id 1bd56ff56fa2: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
  12s           12s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Created         Created container with docker id 960a9c38f1d3
  12s           12s             1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Started         Started container with docker id 960a9c38f1d3
  52s           2s              6       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Warning         Unhealthy       Readiness probe failed: Get http://172.16.43.10:5000/v2/: dial tcp 172.16.43.10:5000: connection refused
  52s           2s              6       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Warning         Unhealthy       Liveness probe failed: Get http://172.16.43.10:5000/v2/: dial tcp 172.16.43.10:5000: connection refused
  2s            2s              1       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Normal          Killing         Killing container with docker id 960a9c38f1d3: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
  2s            1s              2       {kubelet 10.0.2.4}      spec.containers{deis-registry}  Warning         BackOff         Back-off restarting failed docker container
  2s            1s              2       {kubelet 10.0.2.4}                                      Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "deis-registry" with CrashLoopBackOff: "Back-off 20s restarting failed container=deis-registry pod=deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)"
kmala commented 8 years ago

yes..it seems so....can you delete registry pod and get the logs immediatly because sometime the kubectl logs doesn't show full logs when its in crashloopback

monaka commented 8 years ago

I'm keeping inspection. Possibly it is caused lower layer than Registry.

monaka commented 8 years ago

It's my mistake. DNS service in Kube-system was down. After fixing it, Registry was up. I close this issue.

2016/06/20 04:53:56 INFO: Starting registry...
2016/06/20 04:53:56 INFO: using azure as the backend
time="2016-06-20T04:53:57.120359633Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the  REGISTRY_HTTP_SECRET environment variable." go.version=go1.6.1 instance.id=bc90833f-d224-44d0-bad5-d57d6e1856af service=registry version=v2.1.1-497-g5e84d4f 
time="2016-06-20T04:53:57.120443037Z" level=info msg="redis not configured" go.version=go1.6.1 instance.id=bc90833f-d224-44d0-bad5-d57d6e1856af service=registry version=v2.1.1-497-g5e84d4f
(snip)