search

kubesphere / ks-installer

Install KubeSphere on existing Kubernetes cluster
https://kubesphere.io
Apache License 2.0
532 stars 746 forks source link

ImagePullBackOff #70

Closed liuxu623 closed 5 years ago

liuxu623 commented 5 years ago

Some pods in kubesphere-monitoring-system not ready because couldn't pull images from dockerhub.qingcloud.com.

kubectl get pod -n kubesphere-monitoring-system

NAME                                   READY   STATUS             RESTARTS   AGE
kube-state-metrics-55cdd5b576-sjv4r    4/4     Running            0          60m
kube-state-metrics-74597f476b-9c64h    2/4     ImagePullBackOff   0          58m
node-exporter-47ttm                    2/2     Running            0          60m
node-exporter-48zhk                    2/2     Running            0          60m
node-exporter-4pm6p                    2/2     Running            0          60m
node-exporter-5xmkd                    1/2     ImagePullBackOff   0          60m
node-exporter-6242c                    2/2     Running            0          60m
node-exporter-62c6z                    2/2     Running            0          60m
node-exporter-6nq6r                    2/2     Running            0          60m
node-exporter-7fzpz                    2/2     Running            0          60m
node-exporter-7hdn4                    2/2     Running            0          60m
node-exporter-7qmzb                    2/2     Running            0          60m
node-exporter-84fcz                    2/2     Running            0          60m
node-exporter-8j5lw                    2/2     Running            0          60m
node-exporter-97skr                    2/2     Running            0          60m
node-exporter-9ff5w                    2/2     Running            0          60m
node-exporter-9qrth                    2/2     Running            0          60m
node-exporter-9rjr5                    2/2     Running            0          60m
node-exporter-9s4vz                    2/2     Running            0          60m
node-exporter-9st48                    2/2     Running            0          60m
node-exporter-9t2jm                    2/2     Running            0          60m
node-exporter-bb9mh                    2/2     Running            0          60m
node-exporter-bq5xt                    2/2     Running            0          60m
node-exporter-c4vrj                    2/2     Running            0          60m
node-exporter-cr75t                    2/2     Running            0          60m
node-exporter-d2lxp                    2/2     Running            0          60m
node-exporter-d6fh9                    2/2     Running            0          60m
node-exporter-f5kbc                    2/2     Running            0          60m
node-exporter-f6rld                    2/2     Running            0          60m
node-exporter-ftrmb                    2/2     Running            0          60m
node-exporter-fvfc7                    2/2     Running            0          60m
node-exporter-g6m24                    2/2     Running            0          60m
node-exporter-gfpsc                    2/2     Running            0          60m
node-exporter-hf8dp                    2/2     Running            0          60m
node-exporter-hnnl5                    2/2     Running            0          60m
node-exporter-hnzc5                    2/2     Running            0          60m
node-exporter-hz8xj                    2/2     Running            0          60m
node-exporter-j75dk                    2/2     Running            0          60m
node-exporter-jbzh2                    2/2     Running            0          60m
node-exporter-jmt4r                    2/2     Running            0          60m
node-exporter-k5ff9                    2/2     Running            0          60m
node-exporter-klcbc                    2/2     Running            0          60m
node-exporter-krzd2                    2/2     Running            0          60m
node-exporter-lvgsq                    2/2     Running            0          60m
node-exporter-lvj7n                    2/2     Running            0          60m
node-exporter-m6bd9                    2/2     Running            0          60m
node-exporter-m6jlt                    2/2     Running            0          60m
node-exporter-m7zsd                    2/2     Running            0          60m
node-exporter-m9fvx                    2/2     Running            0          60m
node-exporter-mgxhp                    2/2     Running            0          60m
node-exporter-mj49z                    2/2     Running            0          60m
node-exporter-mrsch                    2/2     Running            0          60m
node-exporter-mxg6c                    2/2     Running            0          60m
node-exporter-ngfrd                    2/2     Running            0          60m
node-exporter-nxn2l                    2/2     Running            0          60m
node-exporter-p66c2                    2/2     Running            0          60m
node-exporter-p78h9                    2/2     Running            0          60m
node-exporter-p87wh                    2/2     Running            0          60m
node-exporter-p8wfk                    2/2     Running            0          60m
node-exporter-pb95p                    2/2     Running            0          60m
node-exporter-pmvnj                    2/2     Running            0          60m
node-exporter-pvz2m                    2/2     Running            0          60m
node-exporter-pxt8g                    2/2     Running            0          60m
node-exporter-q4rgp                    2/2     Running            0          60m
node-exporter-qpxwk                    2/2     Running            0          60m
node-exporter-r6kkh                    2/2     Running            0          60m
node-exporter-rk2c4                    2/2     Running            0          60m
node-exporter-rtxgv                    2/2     Running            0          60m
node-exporter-s97jw                    2/2     Running            0          60m
node-exporter-sjqlv                    2/2     Running            0          60m
node-exporter-sjsbc                    2/2     Running            0          60m
node-exporter-tp547                    2/2     Running            0          60m
node-exporter-tpjms                    2/2     Running            0          60m
node-exporter-tzx2l                    2/2     Running            0          60m
node-exporter-vg9p2                    2/2     Running            0          60m
node-exporter-vnffm                    2/2     Running            0          60m
node-exporter-vq2kp                    2/2     Running            0          60m
node-exporter-vq78d                    2/2     Running            0          60m
node-exporter-w5lfc                    2/2     Running            0          60m
node-exporter-wqjll                    2/2     Running            0          60m
node-exporter-x6tmd                    2/2     Running            0          60m
node-exporter-xk7m5                    2/2     Running            0          60m
node-exporter-xp7f9                    2/2     Running            0          60m
node-exporter-xprp5                    2/2     Running            0          60m
node-exporter-xqk77                    2/2     Running            0          60m
node-exporter-z74gw                    2/2     Running            0          60m
node-exporter-zdddp                    2/2     Running            0          60m
node-exporter-zk4tx                    2/2     Running            0          60m
node-exporter-zlhvc                    2/2     Running            0          60m
node-exporter-zn8zx                    2/2     Running            0          60m
prometheus-k8s-0                       0/3     CrashLoopBackOff   16         60m
prometheus-k8s-1                       0/3     CrashLoopBackOff   16         60m
prometheus-k8s-system-0                0/3     CrashLoopBackOff   16         60m
prometheus-k8s-system-1                0/3     CrashLoopBackOff   16         60m
prometheus-operator-799455496d-6jq4v   1/1     Running            1          60m
kubectl describe pod prometheus-k8s-0 -n kubesphere-monitoring-system

Events:
  Type     Reason   Age                  From             Message
  ----     ------   ----                 ----             -------
  Normal   Pulled   59m                  kubelet, kube05  Successfully pulled image "dockerhub.qingcloud.com/prometheus/prometheus:v2.5.0"
  Normal   Pulling  59m                  kubelet, kube05  Pulling image "dockerhub.qingcloud.com/coreos/prometheus-config-reloader:v0.27.1"
  Normal   Pulled   59m                  kubelet, kube05  Container image "dockerhub.qingcloud.com/prometheus/prometheus:v2.5.0" already present on machine
  Warning  Failed   59m                  kubelet, kube05  Error: ErrImagePull
  Warning  Failed   59m                  kubelet, kube05  Failed to pull image "dockerhub.qingcloud.com/coreos/prometheus-config-reloader:v0.27.1": rpc error: code = Unknown desc = Error response from daemon: Get https://dockerhub.qingcloud.com/v2/coreos/prometheus-config-reloader/manifests/v0.27.1: error parsing HTTP 403 response body: no error details found in HTTP response body: "{\"err\":\"user is not active\"}\n"
  Warning  Failed   59m                  kubelet, kube05  Error: ErrImagePull
  Normal   Pulling  59m                  kubelet, kube05  Pulling image "dockerhub.qingcloud.com/coreos/configmap-reload:v0.0.1"
  Warning  Failed   59m                  kubelet, kube05  Failed to pull image "dockerhub.qingcloud.com/coreos/configmap-reload:v0.0.1": rpc error: code = Unknown desc = Error response from daemon: Get https://dockerhub.qingcloud.com/v2/coreos/configmap-reload/manifests/v0.0.1: error parsing HTTP 403 response body: no error details found in HTTP response body: "{\"err\":\"user is not active\"}\n"
  Normal   Started  59m (x2 over 59m)    kubelet, kube05  Started container prometheus
  Normal   Created  59m (x2 over 59m)    kubelet, kube05  Created container prometheus
  Normal   BackOff  59m (x2 over 59m)    kubelet, kube05  Back-off pulling image "dockerhub.qingcloud.com/coreos/configmap-reload:v0.0.1"
  Warning  Failed   59m (x2 over 59m)    kubelet, kube05  Error: ImagePullBackOff
  Normal   BackOff  59m (x3 over 59m)    kubelet, kube05  Back-off pulling image "dockerhub.qingcloud.com/coreos/prometheus-config-reloader:v0.27.1"
  Warning  Failed   59m (x3 over 59m)    kubelet, kube05  Error: ImagePullBackOff
  Warning  BackOff  47s (x285 over 59m)  kubelet, kube05  Back-off restarting failed container

And I try docker login with guest/guest, login success but pull error is user is not active.

$ docker login dockerhub.qingcloud.com

Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

$ docker pull dockerhub.qingcloud.com/coreos/configmap-reload:v0.0.1
Error response from daemon: Get https://dockerhub.qingcloud.com/v2/coreos/configmap-reload/manifests/v0.0.1: error parsing HTTP 403 response body: no error details found in HTTP response body: "{\"err\":\"user is not active\"}\n"
FeynmanZhou commented 5 years ago

@liuxu623 It seems to be something wrong with the QingCloud DockerHub account, it has been fixed already.