search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 670 forks source link

weave-net POD status Error: ImagePullBackOff #3917

Open santhiinvsp opened 3 years ago

santhiinvsp commented 3 years ago

Hi,

I have installed kubeadm and trying to install Weave Net. But weave-net POD is not running.

[root@KubeadmMaster1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-78fcd69978-lg4lc 0/1 Pending 0 118m coredns-78fcd69978-pl2zf 0/1 Pending 0 118m weave-net-wh5pv 0/2 Init:ImagePullBackOff 0 9m4s

I see the following error in describe POD.

Events: Type Reason Age From Message

Normal Scheduled 98s default-scheduler Successfully assigned kube-system/weave-net-wh5pv to kubeadmmaster1.fyre.ibm.com Normal BackOff 14s (x5 over 95s) kubelet Back-off pulling image "docker.io/weaveworks/weave-kube:2.8.1" Warning Failed 14s (x5 over 95s) kubelet Error: ImagePullBackOff Normal Pulling 2s (x4 over 98s) kubelet Pulling image "docker.io/weaveworks/weave-kube:2.8.1" Warning Failed 1s (x4 over 96s) kubelet Failed to pull image "docker.io/weaveworks/weave-kube:2.8.1": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Warning Failed 1s (x4 over 96s) kubelet Error: ErrImagePull

Infact my container image pull image limit from my docker hub is still not reached.

curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 ratelimit-limit: 200;w=21600 ratelimit-remaining: 200;w=21600

I ran docker login, tried to create a secret and then install weave-net using both the following ways but no luck.

1. kubectl create secret generic regcred -n kube-system --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&password-secret=regcred"

2. echo "" > /var/lib/weave/weave-passwd kubectl create secret -n kube-system generic weave-passwd --from-file=/var/lib/weave/weave-passwd kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&password-secret=weave-passwd"

Kubectl version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

How to fix this? How to make weave-net use my dockerhub credentials before installtion?

Thanks and regards Santhi Kumar

santhiinvsp commented 3 years ago

One more thing to add. If pull images(weave-npc:2.8.1, weave-kube:2.8.1) manually using docker pull, then weave-net pod is running.

santhiinvsp commented 3 years ago

Not only this, if I run "kubectl run redis --image=redis", this is also giving me the same docker container image pull limit. How to fix this issue?

Events: Type Reason Age From Message

Normal Pulling 29s (x4 over 119s) kubelet Pulling image "redis" Warning Failed 28s (x4 over 118s) kubelet Failed to pull image "redis": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Warning Failed 28s (x4 over 118s) kubelet Error: ErrImagePull Warning Failed 16s (x6 over 117s) kubelet Error: ImagePullBackOff Normal BackOff 1s (x7 over 117s) kubelet Back-off pulling image "redis"

varkey commented 3 years ago

This doesn't appear to be Weave related though, https://www.docker.com/increase-rate-limit might be more helpful. It is possible that you are pulling weave and redis images without setting the image pull secrets so the images might be getting pulled as an anonymous user thereby hitting the limit.

santhiinvsp commented 3 years ago

This doesn't appear to be Weave related though, https://www.docker.com/increase-rate-limit might be more helpful. It is possible that you are pulling weave and redis images without setting the image pull secrets so the images might be getting pulled as an anonymous user thereby hitting the limit.

@varkey : docker login followed by docker pull is working fine without any issues. I am not sure what's wrong with K8s docker images pull and why its throwing this limit issues. Infact I tried with creating secrets with the following ways, still didn't help.

  1. kubectl create secret generic regcred -n kube-system --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&password-secret=regcred"

  2. echo "" > /var/lib/weave/weave-passwd kubectl create secret -n kube-system generic weave-passwd --from-file=/var/lib/weave/weave-passwd kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&password-secret=weave-passwd"

minhtuyenvp02 commented 5 months ago

I get the same problems. How did you solve it?