search

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
https://litmuschaos.io
Apache License 2.0
4.37k stars 687 forks source link

toomanyrequests: You have reached your pull rate limit #2846

Closed ntlzthm8 closed 3 years ago

ntlzthm8 commented 3 years ago

Question

I try to install litmus operator on minikube three days... apply yuml done ok, but images can't pulled from docker hub

Failed to pull image "litmuschaos/chaos-operator:1.13.5": rpc error: code = Unknown desc = toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

docker login is well (i have account on docker hub)

docker pull - it's pull images without any errors

anyway in docker hub The rate limits of 100 container image requests per six hours for anonymous usage I try install it three days, it's 12 times for 6 hours

I created secret regcred from instruction https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ And it doesn't work I tried to change the yaml file and substitute

    spec:
      containers:
        - name: chaos-operator
          image: docker.io/litmuschaos/chaos-operator:1.13.5
      imagePullSecrets:
        - name: regcred

But it doesn't work, not auth

Portal images pulled ok, after i do docker pull but I don't think it helped

what i do wrong???

ntlzthm8 commented 3 years ago

docker hub curent limits for my account ratelimit-limit: 200;w=21600 ratelimit-remaining: 200;w=21600

oumkale commented 3 years ago

Hi @TidalPoo, Thank you, but this issue still there? This is because mainly due to limit reached, you may delete cluster and try again this issue should be resolve. For more about user : please refer here

ntlzthm8 commented 3 years ago

I known about rate limits, in my account i have 198 pulls ratelimit-limit: 200;w=21600 ratelimit-remaining: 198;w=21600

i'am add addon registry-creds to my minikube

[minikubik@chaostest ~]$ kubectl get secrets --namespace=kube-system | grep registry-creds
registry-creds-acr                               Opaque                                3      36m
registry-creds-dpr                               Opaque                                3      36m
registry-creds-ecr                               Opaque                                6      36m
registry-creds-gcr                               Opaque                                2      36m

add line to yaml

    spec:
      imagePullSecrets:
        - name: registry-creds-dpr
      serviceAccountName: litmus
      containers:
        - name: chaos-operator
          image: litmuschaos/chaos-operator:1.13.5
          command:
          - chaos-operator
          imagePullPolicy: Always
          env:
            - name: CHAOS_RUNNER_IMAGE
              value: "litmuschaos/chaos-runner:1.13.5"
            - name: WATCH_NAMESPACE
              value: ""
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "chaos-operator"

but steel have error

  Warning  Failed     2m10s (x3 over 3m28s)  kubelet            Failed to pull image "litmuschaos/chaos-operator:1.13.5": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   BackOff    115s (x6 over 3m46s)   kubelet            Back-off pulling image "litmuschaos/chaos-operator:1.13.5"
  Warning  Failed     102s (x7 over 3m46s)   kubelet            Error: ImagePullBackOff
imrajdas commented 3 years ago

Hi @TidalPoo, If you're trying it on minikube. I would suggest to ssh minikube node and do a docker login before applying the manifest.

1. minikube ssh
2. docker login
3. kubectl apply -f https://litmuschaos.github.io/litmus/2.0.0-Beta/litmus-2.0.0-Beta.yaml
ntlzthm8 commented 3 years ago

kubectl apply -f https://litmuschaos.github.io/litmus/2.0.0-Beta/litmus-2.0.0-Beta.yaml

i don't need portal, it's work ok, pulled and started. I need operator

  1. minikube ssh
  2. docker login
  3. kubectl apply

it's not working

[minikubik@chaostest ~]$ minikube ssh
Last login: Mon May 31 08:32:56 2021 from 192.168.49.1
docker@minikube:~$ docker login
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/docker/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
docker@minikube:~$ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Mon, 31 May 2021 08:45:12 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 200;w=21600
ratelimit-remaining: 199;w=21600

docker@minikube:~$ docker pull mysql
Using default tag: latest
latest: Pulling from library/mysql
69692152171a: Pull complete
1651b0be3df3: Pull complete
951da7386bc8: Pull complete
0f86c95aa242: Pull complete
37ba2d8bd4fe: Pull complete
6d278bb05e94: Pull complete
497efbd93a3e: Pull complete
f7fddf10c2c2: Pull complete
16415d159dfb: Pull complete
0e530ffc6b73: Pull complete
b0a4a1a77178: Pull complete
cd90f92aa9ef: Pull complete
Digest: sha256:d50098d7fcb25b1fcb24e2d3247cae3fc55815d64fec640dc395840f8fa80969
Status: Downloaded newer image for mysql:latest
docker.io/library/mysql:latest

docker@minikube:~$ exit
logout
[minikubik@chaostest ~]$ minikube image list
k8s.gcr.io/pause:3.2
k8s.gcr.io/kube-scheduler:v1.20.2
k8s.gcr.io/kube-proxy:v1.20.2
k8s.gcr.io/kube-controller-manager:v1.20.2
k8s.gcr.io/kube-apiserver:v1.20.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/echoserver:1.10
k8s.gcr.io/coredns:1.7.0
gcr.io/k8s-minikube/storage-provisioner:v5
gcr.io/google_containers/echoserver:1.4
docker.io/upmcenterprises/registry-creds:<none>
docker.io/litmuschaos/mongo:4.2.8
docker.io/litmuschaos/litmusportal-server:2.0.0-Beta7
docker.io/litmuschaos/litmusportal-frontend:2.0.0-Beta7
docker.io/litmuschaos/litmusportal-auth-server:2.0.0-Beta7
docker.io/litmuschaos/chaos-runner:latest
docker.io/litmuschaos/chaos-runner:1.13.5
docker.io/litmuschaos/chaos-operator:latest
docker.io/litmuschaos/chaos-operator:1.13.5
docker.io/kubernetesui/metrics-scraper:v1.0.4
docker.io/kubernetesui/dashboard:v2.1.0
[minikubik@chaostest ~]$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                        READY   STATUS             RESTARTS   AGE
default                hello-minikube-5d9b964bfb-49k2t             1/1     Running            17         2d20h
kube-system            coredns-74ff55c5b-f652z                     1/1     Running            18         2d20h
kube-system            etcd-minikube                               1/1     Running            18         2d20h
kube-system            kube-apiserver-minikube                     1/1     Running            18         2d20h
kube-system            kube-controller-manager-minikube            1/1     Running            18         2d20h
kube-system            kube-proxy-qgc2l                            1/1     Running            18         2d20h
kube-system            kube-scheduler-minikube                     1/1     Running            18         2d20h
kube-system            registry-creds-85b974c7d7-qshzb             1/1     Running            7          25h
kube-system            storage-provisioner                         1/1     Running            34         2d20h
kubernetes-dashboard   dashboard-metrics-scraper-f6647bd8c-jxr8n   1/1     Running            18         2d20h
kubernetes-dashboard   kubernetes-dashboard-968bcb79-vgzhr         1/1     Running            36         2d20h
litmus                 chaos-operator-ce-5ffd8d8c8b-mtwnx          0/1     ImagePullBackOff   0          6m23s
litmus                 litmusportal-frontend-698bcb686f-ld987      1/1     Running            0          10s
litmus                 litmusportal-server-5bb94f65d7-v84m6        2/2     Running            0          10s
litmus                 mongo-0                                     1/1     Running            0          9s
ntlzthm8 commented 3 years ago

now it's worked changed imagePullPolicy: Always to imagePullPolicy: IfNotPresent