If load balancer IP is (temporarily) unavailable, skupper-router pod can fail to start (due to failed mounts)

[Karen-Schoener]

Describe the bug

If the load balancer is (temporarily) unavailable when a declarative site is created, the site does not transition to a running state.

How To Reproduce

• Using minikube, I ran the the declarative example with skupper v1.
• With minikube tunnel running, I started the site-controller.
• I stopped minikube tunnel, to simulate an intermittent load balancer issue.
• With no minikube tunnel running, created an interior site.
• At this point, the skupper-router pod fails to start (due to failed mounts).
• Wait ~2 minutes for skupper-site-controller to timeout on the step: Initialising skupper site ... Snippet from skupper-site-controller logs:

  2024/09/06 17:40:46 Initialising skupper site ...
  context deadline exceeded
  2024/09/06 17:42:47 Error initialising skupper:  Failed to get LoadBalancer IP or Hostname for service skupper-router
  E0906 17:42:47.047756       1 controller.go:204] Failed to get LoadBalancer IP or Hostname for service skupper-router

• Started minikube tunnel.
• At this point, the site does not appear to recover to a running state.

Expected behavior

After minikube tunnel was started, expected the site to recover to a running state.

Environment details

• Skupper CLI: 1.8.0
• Skupper Operator (if applicable): 1.8.0
• Platform: minikube

[Karen-Schoener]

Closing issue. It's not as reproducible as I thought. Will retrace my steps.

[Karen-Schoener]

Issue is reproducible with the test step:

• Wait ~2 minutes for skupper-site-controller to timeout on the step: Initialising skupper site ...

While in this state, the skupper-router pods remain in a state of ContainerCreating.

 $ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS              RESTARTS      AGE
east          skupper-router-9bb8848f8-x6ght            0/2     ContainerCreating   0             9m28s
east          skupper-site-controller-77f45fc7d-mzhk2   1/1     Running             0             11m
kube-system   coredns-7db6d8ff4d-xt962                  1/1     Running             0             44m
kube-system   etcd-minikube                             1/1     Running             0             44m
kube-system   kube-apiserver-minikube                   1/1     Running             0             44m
kube-system   kube-controller-manager-minikube          1/1     Running             0             44m
kube-system   kube-proxy-rr42v                          1/1     Running             0             44m
kube-system   kube-scheduler-minikube                   1/1     Running             0             44m
kube-system   storage-provisioner                       1/1     Running             1 (43m ago)   44m
west          skupper-router-66bb77c574-9qn29           0/2     ContainerCreating   0             9m33s
west          skupper-site-controller-77f45fc7d-4jshd   1/1     Running             0             11m

While in this state, here's a list of the created secrets.

$ kubectl get secrets -A
NAMESPACE     NAME                     TYPE                            DATA   AGE
east          skupper-console-users    Opaque                          1      13m
east          skupper-local-ca         kubernetes.io/tls               2      13m
east          skupper-local-client     kubernetes.io/tls               4      13m
east          skupper-local-server     kubernetes.io/tls               3      13m
east          skupper-service-ca       kubernetes.io/tls               2      13m
east          skupper-service-client   Opaque                          1      13m
east          skupper-site-ca          kubernetes.io/tls               2      13m
kube-system   bootstrap-token-gnm68z   bootstrap.kubernetes.io/token   6      48m
west          skupper-console-users    Opaque                          1      13m
west          skupper-local-ca         kubernetes.io/tls               2      13m
west          skupper-local-client     kubernetes.io/tls               4      13m
west          skupper-local-server     kubernetes.io/tls               3      13m
west          skupper-service-ca       kubernetes.io/tls               2      13m
west          skupper-service-client   Opaque                          1      13m
west          skupper-site-ca          kubernetes.io/tls               2      13m

Attaching site-controller logs for namespaces west, east.

logs_west_skupper-site-controller.txt logs_east_skupper-site-controller.txt

Attaching kubectl describe pod skupper-router output.

kube_east_describe_pod_skupper_router.txt kube_west_describe_pod_skupper_router.txt