search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
16.21k stars 6.5k forks source link

In offline construction with "ingress_nginx_webhook_enabled: true" The Pod of the Job created when is ImagePullBackOff and does not become READY. #11591

Open DearJey opened 1 month ago

DearJey commented 1 month ago

What happened?

I created Kubernetes Cluster In offline construction with "ingress_nginx_webhook_enabled: true", But "ingress-nginx-admission-*" Pod of the Job created when is ImagePullBackOff and does not become READY.

I checked pod, it was failing pull image "ingress-nginx/kube-webhook-certgen:v1.4.1".

I make list of files and images was created with ./generate_list.sh and registered in the Nginx container with ./manage-offline-files.sh. I then downloaded the container image with manage-offline-container-images.sh and registered it in the local registry. https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/offline/README.md

When I checked the image list created with ./generate_list.sh, the image "ingress-nginx/kube-webhook-certgen:v1.4.1" was not listed.

In my opinion, as a result, the image was not downloaded and the image was not registered in the local registry because of "kube-webhook-certgen:v1.4.1" image was not listed. so it was not possible to register the image in the Job's Pod, and the image became ImagePullBackOff and did not become READY.

What did you expect to happen?

I expected that the image kube-webhook-certgen:v1.4.1 would be pulled to the ingress-nginx-admission of the "ingress-nginx-admission-*" Pod, the Pod would be created, and the Job would complete.

How can we reproduce it (as minimally and precisely as possible)?

[Node configuration] controlplane1 worker1 workstation (Preparation Node:Online)

[Version] Kubespray Version: v2.25.0 Docker version: v26.1

[addons.yml]

ingress_nginx_enabled: true
ingress_nginx_webhook_enabled: true

OS

Linux 5.14.0-362.8.1.el9_3.x86_64 x86_64 NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.3" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

Version of Ansible

ansible [core 2.16.11] config file = None configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.11/site-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.11.5 (main, Sep 7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3.11) jinja version = 3.1.4 libyaml = True

Version of Python

Python 3.9.18

Version of Kubespray (commit)

v2.25.0

Network plugin used

calico

Full inventory with variables

controlplane1 | SUCCESS => { "hostvars[inventory_hostname]": { "allow_unsupported_distribution_setup": false, "ansible_check_mode": false, "ansible_config_file": "/root/k8s-upgrade/kubespray-2.25.0/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 5, "ansible_host": "192.168.122.111", "ansible_inventory_sources": [ "/root/k8s-upgrade/kubespray-2.25.0/inventory/mycluster/inventory.ini" ], "ansible_playbook_python": "/usr/bin/python3.11", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.11", "major": 2, "minor": 16, "revision": 11, "string": "2.16.11" }, "argocd_enabled": false, "auto_renew_certificates": false, "bin_dir": "/usr/local/bin", "calico_cni_name": "k8s-pod-network", "calico_crds_download_url": "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz", "calico_pool_blocksize": 26, "calicoctl_download_url": "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}", "cephfs_provisioner_enabled": false, "cert_manager_enabled": false, "cilium_l2announcements": false, "ciliumcli_download_url": "{{ files_repo }}/github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_version }}/cilium-linux-{{ image_arch }}.tar.gz", "cluster_name": "cluster.local", "cni_download_url": "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz", "container_manager": "containerd", "containerd_download_url": "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz", "containerd_registries_mirrors": [ { "mirrors": [ { "capabilities": [ "pull", "resolve" ], "host": "http://192.168.122.155:5000", "skip_verify": true } ], "prefix": "192.168.122.155:5000" } ], "coredns_k8s_external_zone": "k8s_external.local", "credentials_dir": "/root/k8s-upgrade/kubespray-2.25.0/inventory/mycluster/credentials", "cri_dockerd_download_url": "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/v{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}.{{ image_arch }}.tgz", "crictl_download_url": "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz", "crio_download_url": "{{ files_repo }}/storage.googleapis.com/cri-o/artifacts/cri-o.{{ image_arch }}.{{ crio_version }}.tar.gz", "crun_download_url": "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}", "default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir", "deploy_netchecker": false, "dns_domain": "cluster.local", "dns_mode": "coredns", "docker_bin_dir": "/usr/bin", "docker_container_storage_setup": false, "docker_daemon_graph": "/var/lib/docker", "docker_dns_servers_strict": false, "docker_image_repo": "192.168.122.155:5000", "docker_iptables_enabled": "false", "docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5", "docker_rpm_keepcache": 1, "enable_coredns_k8s_endpoint_pod_names": false, "enable_coredns_k8s_external": false, "enable_dual_stack_networks": false, "enable_nat_default_gateway": true, "enable_nodelocaldns": true, "enable_nodelocaldns_secondary": false, "etcd_data_dir": "/var/lib/etcd", "etcd_deployment_type": "host", "etcd_download_url": "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz", "event_ttl_duration": "1h0m0s", "files_repo": "http://192.168.122.155:8080", "gcr_image_repo": "192.168.122.155:5000", "group_names": [ "etcd", "k8s_cluster", "kube_control_plane" ], "groups": { "all": [ "controlplane1", "worker1" ], "calico_rr": [], "etcd": [ "controlplane1" ], "k8s_cluster": [ "controlplane1", "worker1" ], "kube_control_plane": [ "controlplane1" ], "kube_node": [ "worker1" ], "ungrouped": [] }, "gvisor_containerd_shim_runsc_download_url": "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/containerd-shim-runsc-v1", "gvisor_runsc_download_url": "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/runsc", "helm_download_url": "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz", "helm_enabled": false, "ingress_alb_enabled": false, "ingress_nginx_enabled": true, "ingress_nginx_webhook_enabled": true, "ingress_publish_status_address": "", "inventory_dir": "/root/k8s-upgrade/kubespray-2.25.0/inventory/mycluster", "inventory_file": "/root/k8s-upgrade/kubespray-2.25.0/inventory/mycluster/inventory.ini", "inventory_hostname": "controlplane1", "inventory_hostname_short": "controlplane1", "k8s_image_pull_policy": "IfNotPresent", "kata_containers_download_url": "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz", "kata_containers_enabled": false, "krew_download_url": "{{ files_repo }}/github.com/kubernetes-sigs/krew/releases/download/{{ krew_version }}/krew-{{ hostos }}{{ image_arch }}.tar.gz", "krew_enabled": false, "krew_root_dir": "/usr/local/krew", "kube_api_anonymous_auth": true, "kube_apiserver_ip": "10.233.0.1", "kube_apiserver_port": 6443, "kube_cert_dir": "/etc/kubernetes/ssl", "kube_cert_group": "kube-cert", "kube_config_dir": "/etc/kubernetes", "kube_encrypt_secret_data": false, "kube_image_repo": "192.168.122.155:5000", "kube_log_level": 2, "kube_manifest_dir": "/etc/kubernetes/manifests", "kube_network_node_prefix": 24, "kube_network_node_prefix_ipv6": 120, "kube_network_plugin": "calico", "kube_network_plugin_multus": false, "kube_ovn_default_gateway_check": true, "kube_ovn_default_logical_gateway": false, "kube_ovn_default_vlan_id": 100, "kube_ovn_dpdk_enabled": false, "kube_ovn_enable_external_vpc": true, "kube_ovn_enable_lb": true, "kube_ovn_enable_np": true, "kube_ovn_enable_ssl": false, "kube_ovn_encap_checksum": true, "kube_ovn_external_address": "8.8.8.8", "kube_ovn_external_address_ipv6": "2400:3200::1", "kube_ovn_external_dns": "alauda.cn", "kube_ovn_hw_offload": false, "kube_ovn_ic_autoroute": true, "kube_ovn_ic_dbhost": "127.0.0.1", "kube_ovn_ic_enable": false, "kube_ovn_ic_zone": "kubernetes", "kube_ovn_network_type": "geneve", "kube_ovn_node_switch_cidr": "100.64.0.0/16", "kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64", "kube_ovn_pod_nic_type": "veth_pair", "kube_ovn_traffic_mirror": false, "kube_ovn_tunnel_type": "geneve", "kube_ovn_vlan_name": "product", "kube_owner": "kube", "kube_pods_subnet": "10.233.64.0/18", "kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112", "kube_proxy_mode": "ipvs", "kube_proxy_nodeport_addresses": [], "kube_proxy_strict_arp": false, "kube_script_dir": "/usr/local/bin/kubernetes-scripts", "kube_service_addresses": "10.233.0.0/18", "kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116", "kube_token_dir": "/etc/kubernetes/tokens", "kube_version": "v1.29.5", "kube_vip_enabled": false, "kube_webhook_token_auth": false, "kube_webhook_token_auth_url_skip_tls_verify": false, "kubeadm_certificate_key": "deaffd1ece61feacd56b6608c3b8dbd647dfeb5f89da3c8baf65dbece5cbe2d9", "kubeadm_download_url": "{{ files_repo }}/dl.k8s.io/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm", "kubeadm_patches": { "dest_dir": "/etc/kubernetes/patches", "enabled": false, "source_dir": "/root/k8s-upgrade/kubespray-2.25.0/inventory/mycluster/patches" }, "kubectl_download_url": "{{ files_repo }}/dl.k8s.io/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl", "kubelet_download_url": "{{ files_repo }}/dl.k8s.io/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet", "kubernetes_audit": false, "loadbalancer_apiserver_healthcheck_port": 8081, "loadbalancer_apiserver_port": 6443, "local_path_provisioner_enabled": false, "local_release_dir": "/tmp/releases", "local_volume_provisioner_enabled": false, "macvlan_interface": "eth1", "metallb_enabled": false, "metallb_namespace": "metallb-system", "metallb_speaker_enabled": false, "metrics_server_enabled": false, "ndots": 2, "nerdctl_download_url": "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz", "no_proxy": "localhost,10.233.0.1,192.168.122.0/24", "no_proxy_exclude_workers": false, "node_feature_discovery_enabled": false, "nodelocaldns_bind_metrics_host_ip": false, "nodelocaldns_health_port": 9254, "nodelocaldns_ip": "169.254.25.10", "nodelocaldns_second_health_port": 9256, "nodelocaldns_secondary_skew_seconds": 5, "ntp_enabled": false, "ntp_manage_config": false, "ntp_servers": [ "0.pool.ntp.org iburst", "1.pool.ntp.org iburst", "2.pool.ntp.org iburst", "3.pool.ntp.org iburst" ], "omit": "omit_place_holder35334dcd9e690128ea085b95683a9fe3a4607e07", "persistent_volumes_enabled": false, "playbook_dir": "/root/k8s-upgrade/kubespray-2.25.0", "quay_image_repo": "192.168.122.155:5000", "rbd_provisioner_enabled": false, "registry_enabled": false, "registry_host": "192.168.122.155:5000", "remove_anonymous_access": false, "resolvconf_mode": "host_resolvconf", "retry_stagger": 5, "runc_download_url": "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}", "skopeo_download_url": "{{ files_repo }}/github.com/lework/skopeo-binary/releases/download/{{ skopeo_version }}/skopeo-linux-{{ image_arch }}", "skydns_server": "10.233.0.3", "skydns_server_secondary": "10.233.0.4", "unsafe_show_logs": false, "volume_cross_zone_attachment": false } }

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/inventory.ini -b cluster.yml 2>&1 | tee k8s-cluster-install.log

Output of ansible run

controlplane1 : ok=634 changed=139 unreachable=0 failed=0 skipped=1094 rescued=0 ignored=6
worker1 : ok=430 changed=85 unreachable=0 failed=0 skipped=669 rescued=0 ignored=1

Anything else we need to know

No response