What happened?

We always reach the docker rate limit (100 / 6) when we deploy Kubernetes with kubespray. Not sure which default id get used when pull images from docker. We are looking for a possibility to use a private dockerhub account to pull Images from docker.

We are using the kubspray ansible playbook

The solution from issue #6883 is not working anymore because containered_config is depricated.

What did you expect to happen?

That the deployment of kubernetes with a private dockerhub account does not fail when images get pulled from docker.

How can we reproduce it (as minimally and precisely as possible)?

We looking for a possibility to use a private dockerhub account to pull Images from docker

OS

printf "$(uname -srm)\n$(cat /etc/os-release)"

Linux 4.18.0-552.el8_10.x86_64 x86_64

NAME="Red Hat Enterprise Linux" VERSION="8.10 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.10" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.10 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.10"[root@kubetest-ocp-host ~]#

Version of Ansible

ansible --version

ansible [core 2.16.7] config file = None configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.12/site-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.12.1 (main, Feb 21 2024, 10:25:11) [GCC 8.5.0 20210514 (Red Hat 8.5.0-21)] (/usr/bin/python3) jinja version = 3.1.4 libyaml = True

Version of Python

python3 --version Python 3.12.1

Version of Kubespray (commit)

README.md: - kubernetes v1.29.5

Network plugin used

calico

Full inventory with variables

cat inventory/mycluster/inventory.yaml

all: hosts: kube-32: ansible_host: 172.16.3.92 ip: 172.16.3.92 access_ip: 172.16.3.92 kube-33: ansible_host: 172.16.3.93 ip: 172.16.3.93 access_ip: 172.16.3.93 kube-34: ansible_host: 172.16.3.94 ip: 172.16.3.94 access_ip: 172.16.3.94 kube-35: ansible_host: 172.16.3.95 ip: 172.16.3.95 access_ip: 172.16.3.95 kube-36: ansible_host: 172.16.3.96 ip: 172.16.3.96 access_ip: 172.16.3.96 children: kube_control_plane: hosts: kube-32: kube-33: kube_node: hosts: kube-32: kube-33: kube-34: kube-35: kube-36: etcd: hosts: kube-32: kube-33: kube-34: k8s_cluster: children: kube_control_plane: kube_node: calico_rr: hosts: {}

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

Output of ansible run

2024-06-26 14:08:17,320: [I] TASK [download : debug] **** 2024-06-26 14:08:17,320: [I] ok: [kube-32] => { 2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True" 2024-06-26 14:08:17,320: [I] } 2024-06-26 14:08:17,320: [I] ok: [kube-33] => { 2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True" 2024-06-26 14:08:17,320: [I] } 2024-06-26 14:08:17,320: [I] ok: [kube-34] => { 2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True" 2024-06-26 14:08:17,320: [I] } 2024-06-26 14:08:17,320: [I] ok: [kube-35] => { 2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True" 2024-06-26 14:08:17,320: [I] } 2024-06-26 14:08:17,320: [I] ok: [kube-36] => { 2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True" 2024-06-26 14:08:17,320: [I] } 2024-06-26 14:08:52,456: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (4 retries left). 2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (4 retries left). 2024-06-26 14:08:52,457: [I] 2024-06-26 14:08:52,457: [I] TASK [download : Download_container | Download image if required] ** 2024-06-26 14:08:52,457: [I] changed: [kube-33] 2024-06-26 14:08:52,457: [I] changed: [kube-32] 2024-06-26 14:08:52,457: [I] changed: [kube-36] 2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (3 retries left). 2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (2 retries left). 2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (3 retries left). 2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (1 retries left). 2024-06-26 14:08:52,457: [I] fatal: [kube-35]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "docker.io/library/nginx:1.25.2-alpine"], "delta": "0:00:00.803006", "end": "2024-06-26 14:08:36.020879", "msg": "non-zero return code", "rc": 1, "start": "2024-06-26 14:08:35.217873", "stderr": "time=\"2024-06-26T14:08:35+02:00\" level=warning msg=\"DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead.\"\nctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit", "stderr_lines": ["time=\"2024-06-26T14:08:35+02:00\" level=warning msg=\"DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead.\"", "ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.4 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.5 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.6 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.4 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.5 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.6 s total: 0.0 B (0.0 B/s) "]} 2024-06-26 14:08:52,458: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (2 retries left). 2024-06-26 14:08:52,458: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (1 retries left). 2024-06-26 14:08:52,459: [I] fatal: [kube-34]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "docker.io/library/nginx:1.25.2-alpine"], "delta": "0:00:00.867968", "end": "2024-06-26 14:08:52.421105", "msg": "non-zero return code", "rc": 1, "start": "2024-06-26 14:08:51.553137", "stderr": "time=\"2024-06-26T14:08:51+02:00\" level=warning msg=\"DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead.\"\nctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit", "stderr_lines": ["time=\"2024-06-26T14:08:51+02:00\" level=warning msg=\"DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead.\"", "ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.4 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.5 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.6 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.7 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.4 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.5 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.6 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.7 s total: 0.0 B (0.0 B/s) "]} 2024-06-26 14:08:52,460: [I] 2024-06-26 14:08:52,460: [I] NO MORE HOSTS LEFT * 2024-06-26 14:08:52,460: [I] 2024-06-26 14:08:52,460: [I] PLAY RECAP ***** 2024-06-26 14:08:52,460: [I] kube-32 : ok=354 changed=58 unreachable=0 failed=0 skipped=417 rescued=0 ignored=0 2024-06-26 14:08:52,460: [I] kube-33 : ok=324 changed=58 unreachable=0 failed=0 skipped=361 rescued=0 ignored=0 2024-06-26 14:08:52,460: [I] kube-34 : ok=308 changed=55 unreachable=0 failed=1 skipped=358 rescued=0 ignored=0 2024-06-26 14:08:52,460: [I] kube-35 : ok=297 changed=53 unreachable=0 failed=1 skipped=358 rescued=0 ignored=0 2024-06-26 14:08:52,460: [I] kube-36 : ok=298 changed=54 unreachable=0 failed=0 skipped=358 rescued=0 ignored=0 2024-06-26 14:08:52,460: [I] 2024-06-26 14:08:53,599: [E] Command finished with rc = 2

Anything else we need to know

No response

kubernetes-sigs / kubespray

Dockerhub creds for image pulls #11329