Chapter 9 minikube addons enable ingress failure in China

ongiant commented 1 year ago

Hi, I'm stuck in chapter 9.5.1. When I tried to enable ingress, but always failed.

First, I start a minikube cluster with 2 nodes:

cluster details


➜  ~ minikube start -p polar --nodes 2                                                                                                                       
😄  [polar] minikube v1.30.1 on Arch "23.0.0"                                                                                                                
✨  Automatically selected the docker driver                                                                                                                 
📌  Using Docker driver with root privileges                                                                                                                 
👍  Starting control plane node polar in cluster polar                                                                                                       
🚜  Pulling base image ...                                                                                                                                   
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...                                                                                                    
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...                                                                                                        
    ▪ Generating certificates and keys ...                                                                                                                   
    ▪ Booting up control plane ...                                                                                                                           
    ▪ Configuring RBAC rules ...                                                                                                                             
🔗  Configuring CNI (Container Networking Interface) ...                                                                                                     
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5                                                                                                 
🌟  Enabled addons: storage-provisioner, default-storageclass                                                                                                
🔎  Verifying Kubernetes components...                                                                                                                       

👍  Starting worker node polar-m02 in cluster polar                                                                                                          
🚜  Pulling base image ...                                                                                                                                   
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...                                                                                                    
🌐  Found network options:                                                                                                                                   
    ▪ NO_PROXY=192.168.49.2                                                                                                                                  
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...                                                                                                        
    ▪ env NO_PROXY=192.168.49.2                                                                                                                              
🔎  Verifying Kubernetes components...                                                                                                                       
🏄  Done! kubectl is now configured to use "polar" cluster and "default" namespace by default

Then I tried to enable ingress :


➜  ~ minikube addons enable ingress -p polar
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.7.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│    Please also attach the following file to the GitHub issue:                             │
│    - /tmp/minikube_addons_ae4fcb4856619e6ac42e679469959984454bfd00_0.log                  │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

some infomation (These are all collapsible code blocks, you can expand by clicking them):

kubectl get pods -A


➜  ~ kubectl get pods -A                                                      
NAMESPACE       NAME                                        READY   STATUS              RESTARTS      AGE
ingress-nginx   ingress-nginx-admission-create-wkf7d        0/1     ImagePullBackOff    0             16m
ingress-nginx   ingress-nginx-admission-patch-2pqg8         0/1     ImagePullBackOff    0             16m
ingress-nginx   ingress-nginx-controller-6cc5ccb977-q2w47   0/1     ContainerCreating   0             16m
kube-system     coredns-787d4945fb-xjw4m                    1/1     Running             2 (16m ago)   17m
kube-system     etcd-polar                                  1/1     Running             0             17m
kube-system     kindnet-96ksk                               1/1     Running             0             17m
kube-system     kindnet-q8vlj                               1/1     Running             0             17m
kube-system     kube-apiserver-polar                        1/1     Running             0             17m
kube-system     kube-controller-manager-polar               1/1     Running             0             17m
kube-system     kube-proxy-5l2zn                            1/1     Running             0             17m
kube-system     kube-proxy-ng6q8                            1/1     Running             0             17m
kube-system     kube-scheduler-polar                        1/1     Running             0             17m
kube-system     storage-provisioner                         1/1     Running             1 (17m ago)   17m

kubectl describe -n ingress-nginx ingress-nginx-controller-6cc5ccb977-q2w47


➜  ~ kubectl describe pod -n ingress-nginx ingress-nginx-controller-6cc5ccb977-q2w47                                                                                                                                                                                          
Name:             ingress-nginx-controller-6cc5ccb977-q2w47                                                                                                                                                                                                                   
Namespace:        ingress-nginx                                                                                                                                                                                                                                               
Priority:         0                                                                                                                                                                                                                                                           
Service Account:  ingress-nginx                                                                                                                                                                                                                                               
Node:             polar/192.168.49.2                                                                                                                                                                                                                                          
Start Time:       Thu, 13 Jul 2023 10:02:11 +0800                                                                                                                                                                                                                             
Labels:           app.kubernetes.io/component=controller                                                                                                                                                                                                                      
                  app.kubernetes.io/instance=ingress-nginx                                                                                                                                                                                                                    
                  app.kubernetes.io/name=ingress-nginx                                                                                                                                                                                                                        
                  gcp-auth-skip-secret=true                                                                                                                                                                                                                                   
                  pod-template-hash=6cc5ccb977                                                                                                                                                                                                                                
Annotations:                                                                                                                                                                                                                                                            
Status:           Pending                                                                                                                                                                                                                                                     
IP:                                                                                                                                                                                                                                                                           
IPs:                                                                                                                                                                                                                                                                    
Controlled By:    ReplicaSet/ingress-nginx-controller-6cc5ccb977                                                                                                                                                                                                              
Containers:                                                                                                                                                                                                                                                                   
  controller:                                                                                                                                                                                                                                                                 
    Container ID:                                                                                                                                                                                                                                                             
    Image:         registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7                                                                                                                                    
    Image ID:                                                                                                                                                                                                                                                                 
    Ports:         80/TCP, 443/TCP, 8443/TCP                                                                                                                                                                                                                                  
    Host Ports:    80/TCP, 443/TCP, 0/TCP                                                                             
    Args:                            
      /nginx-ingress-controller                                                                                                                              
      --election-id=ingress-nginx-leader                                                                              
      --controller-class=k8s.io/ingress-nginx                                 
      --watch-ingress-without-class=true                                      
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller                                                                            
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services                                                                                                 
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services                                                                                                 
      --validating-webhook=:8443                                              
      --validating-webhook-certificate=/usr/local/certificates/cert                                                                                          
      --validating-webhook-key=/usr/local/certificates/key                                                                                                                                                                                  
    State:          Waiting                                                                                                                                                                                                                 
      Reason:       ContainerCreating                                         
    Ready:          False                                                                                                                                                                                                                   
    Restart Count:  0                                                                                                                                        
    Requests:                                                                 
      cpu:      100m                      
      memory:   90Mi                                                                                                  
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5                                                                                                                                        
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3                                                                                                                                                                          
    Environment:                                                   
      POD_NAME:       ingress-nginx-controller-6cc5ccb977-q2w47 (v1:metadata.name)                                                                                                                                                                                            
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)                                                                            
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so                                                                                    
    Mounts:                                                        
      /usr/local/certificates/ from webhook-cert (ro)                                                                                  
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dxd2k (ro)                                                                                                                                                                                           
Conditions:                                                        
  Type              Status                                         
  Initialized       True                                           
  Ready             False                                          
  ContainersReady   False                                          
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-dxd2k:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    18m                 default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-6cc5ccb977-q2w47 to polar
  Warning  FailedMount  12m (x2 over 16m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube-api-access-dxd2k webhook-cert]: timed out waiting for the condition
  Warning  FailedMount  63s (x6 over 14m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-dxd2k]: timed out waiting for the condition
  Warning  FailedMount  35s (x17 over 18m)  kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found

kubectl describe pod -n ingress-nginx ingress-nginx-admission-create-wkf7d


➜  ~ kubectl describe pod -n ingress-nginx ingress-nginx-admission-create-wkf7d                                                                                                                                                                                               
Name:             ingress-nginx-admission-create-wkf7d                                                                                                                                                                                                                        
Namespace:        ingress-nginx                                                                                                                                                                                                                                               
Priority:         0                                                                                                                                                                                                                                                           
Service Account:  ingress-nginx-admission                                                                                                                                                                                                                                     
Node:             polar/192.168.49.2                                                                                                                                                                                                                                          
Start Time:       Thu, 13 Jul 2023 10:02:11 +0800                                                                                                                                                                                                                             
Labels:           app.kubernetes.io/component=admission-webhook                                                                                                                                                                                                               
                  app.kubernetes.io/instance=ingress-nginx                                                                                                                                                                                                                    
                  app.kubernetes.io/name=ingress-nginx                                                                                                                                                                                                                        
                  controller-uid=d5123061-5639-463a-a4fc-5e73dfa3295d                                                                                                                                                                                                         
                  job-name=ingress-nginx-admission-create                                                                                                                                                                                                                     
Annotations:      
Status:           Pending
IP:               10.244.0.4
IPs:
  IP:           10.244.0.4
Controlled By:  Job/ingress-nginx-admission-create
Containers:
  create:
    Container ID:  
    Image:         registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f
    Image ID:      
    Port:          
    Host Port:     
    Args:
      create
      --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=ingress-nginx-admission
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-82vfm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-82vfm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       22m                   default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-create-wkf7d to polar
  Normal   SandboxChanged  21m                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          19m (x6 over 21m)     kubelet            Error: ImagePullBackOff
  Normal   Pulling         18m (x4 over 22m)     kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"
  Warning  Failed          18m (x4 over 21m)     kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": rpc error: code = Unknown desc = Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": dial tcp 142.251.2.82:443: i/o timeout
  Warning  Failed          18m (x4 over 21m)     kubelet            Error: ErrImagePull
  Warning  Failed          7m26s                 kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": rpc error: code = Unknown desc = Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": dial tcp 74.125.137.82:443: i/o timeout
  Normal   BackOff         2m28s (x69 over 21m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"

kubectl describe pod -n ingress-nginx ingress-nginx-admission-patch-2pqg8


➜  ~ kubectl describe pod -n ingress-nginx ingress-nginx-admission-patch-2pqg8                                                                                                                                                                                        [10/605]
Name:             ingress-nginx-admission-patch-2pqg8                                                                                                                                                                                                                         
Namespace:        ingress-nginx                                                                                                                                                                                                                                               
Priority:         0                                                                                                                                                                                                                                                           
Service Account:  ingress-nginx-admission                                                                                                                                                                                                                                     
Node:             polar/192.168.49.2                                                                                                                                                                                                                                          
Start Time:       Thu, 13 Jul 2023 10:02:11 +0800                                                                                                                                                                                                                             
Labels:           app.kubernetes.io/component=admission-webhook                                                                                                                                                                                                               
                  app.kubernetes.io/instance=ingress-nginx                                                                                                                                                                                                                    
                  app.kubernetes.io/name=ingress-nginx                                                                                                                                                                                                                        
                  controller-uid=200118a6-d7cb-434f-b6e1-013cc284f3e1                                                                                                                                                                                                         
                  job-name=ingress-nginx-admission-patch
Annotations:      
Status:           Pending
IP:               10.244.0.3
IPs:
  IP:           10.244.0.3
Controlled By:  Job/ingress-nginx-admission-patch
Containers:
  patch:
    Container ID:  
    Image:         registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f
    Image ID:      
    Port:          
    Host Port:     
    Args:
      patch
      --webhook-name=ingress-nginx-admission
      --namespace=$(POD_NAMESPACE)
      --patch-mutating=false
      --secret-name=ingress-nginx-admission
      --patch-failure-policy=Fail
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wswbm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-wswbm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                  From               Message
  ----     ------          ----                 ----               -------
  Normal   Scheduled       26m                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-patch-2pqg8 to polar
  Normal   SandboxChanged  26m                  kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          23m (x6 over 26m)    kubelet            Error: ImagePullBackOff
  Warning  Failed          22m (x4 over 26m)    kubelet            Error: ErrImagePull
  Normal   BackOff         11m (x48 over 26m)   kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"
  Warning  Failed          6m33s (x7 over 26m)  kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": rpc error: code = Unknown desc = Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": dial tcp 142.251.2.82:443: i/o timeout
  Normal   Pulling         90s (x9 over 26m)    kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"

log file:
minikube logs --file=logs.txt

logs.txt

abel533 commented 1 year ago

China is unable to download images from registry.k8s.io. You can use a github codespace to download them from here, then upload to your own dockerhub, and download these images locally from dockerhub. Change them to the corresponding names with docker tag and try again.

ongiant commented 1 year ago

China is unable to download images from registry.k8s.io. You can use a github codespace to download them from here, then upload to your own dockerhub, and download these images locally from dockerhub. Change them to the corresponding names with docker tag and try again.

Thanks for your answer, But I have configed a proxy on my local machine and downloaded the image. Afterwards, I used minikube image load command to load the image and attempted to enable ingress, but it still failed. Could there be an issue with my image name and tag?

➜  ~ docker images
REPOSITORY                                           TAG                                        IMAGE ID       CREATED        SIZE
paketobuildpacks/run                                 base-cnb                                   f2e5000af0cb   2 weeks ago    87MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   v20230407                                  7e7451bb7042   3 months ago   47.2MB
kicbase/stable                                       v0.0.39                                    67a4b1138d2d   3 months ago   1.05GB
gcr.io/k8s-minikube/kicbase                          v0.0.39                                    67a4b1138d2d   3 months ago   1.05GB
registry.k8s.io/ingress-nginx/controller             v1.7.0                                     0d4c0564c465   3 months ago   283MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   v20230312-helm-chart-4.5.2-28-g66a760794   5a86b03a88d2   4 months ago   47.1MB
edge-service                                         latest                                     068862751604   43 years ago   297MB
config-service                                       latest                                     9ccc5cc0068a   43 years ago   289MB
order-service                                        latest                                     b91d8d94ccb3   43 years ago   298MB
paketobuildpacks/builder                             base                                       99ec7fb86b9d   43 years ago   1.34GB
catalog-service                                      latest                                     595998cf4202   43 years ago   293MB

ongiant commented 1 year ago

I solved it by changing the image registry. However, I had to set .spec.template.spec.hostNetwork: to true (see:https://github.com/kubernetes/minikube/issues/16828#issuecomment-1716160177 ) in the Deployment of ingress-nginx-controller in order for ingress-nginx-controller to work properly. I would like to know why it can work correctly just by using the minikube addons enable ingress command as mentioned in the book.

ThomasVitale commented 10 months ago

@ongiant thanks for reporting the issue and for sharing a working solution. I have now included a guide for configuring Minikube behind a proxy and linked to your instructions to make the ingress work. Thank you!

I'm closing this issue for now, but feel free to leave new comments in case you find new information about this problem, or open a new issue. It looks like there's some problem in the specific ingress add-on in Minikube, so I hope that will be fixed in the Minikube project to make the setup easier when behind a proxy.

ThomasVitale / cloud-native-spring-in-action

Chapter 9 minikube addons enable ingress failure in China #52