search

mindwm / mindwm-gitops

5 stars 5 forks source link

istiod chart deployment fail #20

Closed metacoma closed 1 month ago

metacoma commented 1 month ago

Possible regression after #18 

one or more objects failed to apply, reason: Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal number into Go struct field ResourceRequirements.spec.template.spec.containers.resources.requests of type v1.ResourceList (retried 5 times).
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  4m57s (x21 over 60m)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "object.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": no endpoints available for service "istiod"

image

metacoma commented 1 month ago

Istio Ingress service never gets a LoadBalancer ip

$ kubectl -n istio-system get svc
NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                      AGE
istiod                  ClusterIP      10.43.225.236   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP        12m
istio-ingressgateway    LoadBalancer   10.43.195.5     <pending>     15021:32139/TCP,80:32395/TCP,443:31617/TCP   10m
knative-local-gateway   ClusterIP      10.43.63.114    <none>        80/TCP,443/TCP                               8m59s
metacoma commented 1 month ago

The root cause is that the servicelb pod for svclb istio ingressgateway container is in a Pending state

$ kubectl get pods --all-namespaces | grep -i svclb-istio-ingressgateway
kube-system         svclb-istio-ingressgateway-9eb93b90-tm9vr                   0/3     Pending     0             27m
kubectl -n kube-system describe pod svclb-istio-ingressgateway-9eb93b90-tm9vr
...
  Warning  FailedScheduling  31m                   default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling  8m20s (x12 over 31m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
metacoma commented 1 month ago

This happens because another kubernetes service (ArgoCD) uses port 443 and has a LoadBalancer type

$ kubectl -n argocd get svc argocd-server
NAME            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                         AGE
argocd-server   LoadBalancer   10.43.39.212   10.20.30.211   38080:31562/TCP,443:30561/TCP   77m
metacoma commented 1 month ago

Fixed by #24 and #21