search

vhive-serverless / vHive

vHive: Open-source framework for serverless experimentation
MIT License
285 stars 86 forks source link

OpenYurt Knative Setup Issue #950

Closed yulinzou closed 2 months ago

yulinzou commented 7 months ago

Describe the bug Failed to setup Knative for OpenYurt, the cluster-local-gateway and istio-ingressgateway pod are not in READY status

- Processing resources for Istio core.
✔ Istio core installed
- Processing resources for Istiod.
- Processing resources for Istiod. Waiting for Deployment/istio-system/istiod
✔ Istiod installed
- Processing resources for Ingress gateways.
- Processing resources for Ingress gateways. Waiting for Deployment/istio-system/cluster-local-ga...
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
  Deployment/istio-system/cluster-local-gateway (containers with unready status: [istio-proxy])
  Deployment/istio-system/istio-ingressgateway (containers with unready status: [istio-proxy])
- Pruning removed resourcesError: failed to install manifests: errors occurred during operation

To Reproduce After setting up Kubernetes cluster include cloud and edge nodes, run ./openyurt_deployer knative, following the OpenYurt setup manual.

Expected behavior Knative should be successfully setup for OpenYurt.

Logs kubectl get pods -n istio-system

NAME                                     READY   STATUS    RESTARTS   AGE
cluster-local-gateway-76bbc4bf78-lnjm2   0/1     Running   0          15m
istio-ingressgateway-dbcbdd6d5-lvw2q     0/1     Running   0          15m
istiod-657b54846b-5sxrq                  1/1     Running   0          15m

kubectl get svc -n istio-system

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                      AGE
cluster-local-gateway   ClusterIP      10.105.191.168   <none>        15020/TCP,80/TCP,443/TCP                     20m
istio-ingressgateway    LoadBalancer   10.97.13.94      <pending>     15021:31926/TCP,80:30778/TCP,443:31594/TCP   20m
istiod                  ClusterIP      10.99.107.159    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP        20m
knative-local-gateway   ClusterIP      10.100.181.234   <none>        80/TCP                                       14m

kubectl logs cluster-local-gateway-76bbc4bf78-lnjm2 -n istio-system --tail=20

2024-03-05T08:39:30.972101Z warn    ca  ca request failed, starting attempt 4 in 797.764779ms
2024-03-05T08:39:31.770157Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:31.912041Z warn    ca  ca request failed, starting attempt 1 in 102.654655ms
2024-03-05T08:39:32.015484Z warn    ca  ca request failed, starting attempt 2 in 203.884035ms
2024-03-05T08:39:32.220005Z warn    ca  ca request failed, starting attempt 3 in 367.516646ms
2024-03-05T08:39:32.588638Z warn    ca  ca request failed, starting attempt 4 in 846.114595ms
2024-03-05T08:39:33.434912Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:33.642198Z warn    ca  ca request failed, starting attempt 1 in 93.76391ms
2024-03-05T08:39:33.736643Z warn    ca  ca request failed, starting attempt 2 in 188.254413ms
2024-03-05T08:39:33.925052Z warn    ca  ca request failed, starting attempt 3 in 430.149563ms
2024-03-05T08:39:34.355720Z warn    ca  ca request failed, starting attempt 4 in 852.894099ms
2024-03-05T08:39:35.209024Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:35.608001Z warn    ca  ca request failed, starting attempt 1 in 94.065982ms
2024-03-05T08:39:35.702460Z warn    ca  ca request failed, starting attempt 2 in 193.463351ms
2024-03-05T08:39:35.896931Z warn    ca  ca request failed, starting attempt 3 in 392.137813ms
2024-03-05T08:39:36.289533Z warn    ca  ca request failed, starting attempt 4 in 871.494812ms
2024-03-05T08:39:37.161684Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:39:37.563544Z warn    ca  ca request failed, starting attempt 1 in 91.317602ms
2024-03-05T08:39:45.811213Z warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 3220s ago: 14, connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"
2024-03-05T08:40:14.606232Z warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 3249s ago: 14, connection error: desc = "transport: Error while dialing dial tcp 10.99.107.159:15012: i/o timeout"

kubectl logs istio-ingressgateway-dbcbdd6d5-lvw2q -n istio-system --tail=20

2024-03-05T08:39:18.676212Z warn    ca  ca request failed, starting attempt 1 in 109.786691ms
2024-03-05T08:39:18.786619Z warn    ca  ca request failed, starting attempt 2 in 215.392111ms
2024-03-05T08:39:19.003146Z warn    ca  ca request failed, starting attempt 3 in 399.242009ms
2024-03-05T08:39:19.267186Z warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 3194s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2024-03-05T08:39:19.402866Z warn    ca  ca request failed, starting attempt 4 in 799.453932ms
2024-03-05T08:39:20.203095Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 192.168.32.2:34056->10.96.0.10:53: i/o timeout"
2024-03-05T08:39:20.340578Z warn    ca  ca request failed, starting attempt 1 in 94.646846ms
2024-03-05T08:39:20.436013Z warn    ca  ca request failed, starting attempt 2 in 219.345007ms
2024-03-05T08:39:20.656524Z warn    ca  ca request failed, starting attempt 3 in 399.759668ms
2024-03-05T08:39:21.057245Z warn    ca  ca request failed, starting attempt 4 in 863.372867ms
2024-03-05T08:39:21.921368Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 192.168.32.2:34056->10.96.0.10:53: i/o timeout"
2024-03-05T08:39:22.142503Z warn    ca  ca request failed, starting attempt 1 in 94.927632ms
2024-03-05T08:39:22.237921Z warn    ca  ca request failed, starting attempt 2 in 206.659364ms
2024-03-05T08:39:22.445396Z warn    ca  ca request failed, starting attempt 3 in 381.600133ms
2024-03-05T08:39:22.828096Z warn    ca  ca request failed, starting attempt 4 in 830.636367ms
2024-03-05T08:39:23.659210Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 192.168.32.2:34056->10.96.0.10:53: i/o timeout"
2024-03-05T08:39:23.966994Z warn    ca  ca request failed, starting attempt 1 in 101.581298ms
2024-03-05T08:39:24.069457Z warn    ca  ca request failed, starting attempt 2 in 189.259981ms
2024-03-05T08:39:24.258852Z warn    ca  ca request failed, starting attempt 3 in 362.50633ms
2024-03-05T08:39:24.622570Z warn    ca  ca request failed, starting attempt 4 in 873.762276ms

Notes The configuration of nodes is as following, two nodes on cloud site, two nodes on edge sides, using image emulab-ops/UBUNTU20-64-STD

{
    "master": "yulin001@hp101.utah.cloudlab.us",
    "workers": {
        "cloud": [
            "yulin001@hp118.utah.cloudlab.us"
        ],
        "edge": [
            "yulin001@hp111.utah.cloudlab.us",
        "yulin001@hp086.utah.cloudlab.us"
        ]
    }
}
yulinzou commented 7 months ago

kubectl describe pod istio-ingressgateway-dbcbdd6d5-gd9jf -n istio-system

Name:             istio-ingressgateway-dbcbdd6d5-gd9jf
Namespace:        istio-system
Priority:         0
Service Account:  istio-ingressgateway-service-account
Node:             edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us/128.110.218.150
Start Time:       Tue, 05 Mar 2024 02:16:53 -0700
Labels:           app=istio-ingressgateway
                  chart=gateways
                  heritage=Tiller
                  install.operator.istio.io/owning-resource=unknown
                  istio=ingressgateway
                  istio.io/rev=default
                  operator.istio.io/component=IngressGateways
                  pod-template-hash=dbcbdd6d5
                  release=istio
                  service.istio.io/canonical-name=istio-ingressgateway
                  service.istio.io/canonical-revision=latest
                  sidecar.istio.io/inject=false
Annotations:      cni.projectcalico.org/containerID: ad4a6fe1c266d4264a17c4c037581461abceec9904bc5253f6cbd562400aac79
                  cni.projectcalico.org/podIP: 192.168.157.194/32
                  cni.projectcalico.org/podIPs: 192.168.157.194/32
                  prometheus.io/path: /stats/prometheus
                  prometheus.io/port: 15020
                  prometheus.io/scrape: true
                  sidecar.istio.io/inject: false
Status:           Running
IP:               192.168.157.194
IPs:
  IP:           192.168.157.194
Controlled By:  ReplicaSet/istio-ingressgateway-dbcbdd6d5
Containers:
  istio-proxy:
    Container ID:  containerd://694a4a8c920e222c07990a2762cb14669a7c05ae1507f02e108e56e8f59c456e
    Image:         docker.io/istio/proxyv2:1.16.3
    Image ID:      docker.io/istio/proxyv2@sha256:35ecc61d241242e8d68746fcccb253c4abc7d3b7671702ddb9e20b532cc514f2
    Ports:         15021/TCP, 8080/TCP, 8443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
    State:          Running
      Started:      Tue, 05 Mar 2024 02:17:00 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                   first-party-jwt
      PILOT_CERT_PROVIDER:          istiod
      CA_ADDR:                      istiod.istio-system.svc:15012
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     istio-ingressgateway-dbcbdd6d5-gd9jf (v1:metadata.name)
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      SERVICE_ACCOUNT:               (v1:spec.serviceAccountName)
      ISTIO_META_WORKLOAD_NAME:     istio-ingressgateway
      ISTIO_META_OWNER:             kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
      ISTIO_META_MESH_ID:           cluster.local
      TRUST_DOMAIN:                 cluster.local
      ISTIO_META_UNPRIVILEGED_POD:  true
      ISTIO_META_CLUSTER_ID:        Kubernetes
    Mounts:
      /etc/istio/config from config-volume (rw)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /etc/istio/pod from podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/credential-uds from credential-socket (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9sggs (ro)
      /var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
      /var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  workload-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  credential-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  workload-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  kube-api-access-9sggs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  9m2s                  default-scheduler  Successfully assigned istio-system/istio-ingressgateway-dbcbdd6d5-gd9jf to edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us
  Normal   Pulling    9m2s                  kubelet            Pulling image "docker.io/istio/proxyv2:1.16.3"
  Normal   Pulled     8m55s                 kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.16.3" in 443.023989ms (6.245443949s including waiting)
  Normal   Created    8m55s                 kubelet            Created container istio-proxy
  Normal   Started    8m55s                 kubelet            Started container istio-proxy
  Warning  Unhealthy  4m (x152 over 8m54s)  kubelet            Readiness probe failed: Get "http://192.168.157.194:15021/healthz/ready": dial tcp 192.168.157.194:15021: connect: connection refused

kubectl describe pod cluster-local-gateway-76bbc4bf78-xmjpv -n istio-system

Name:             cluster-local-gateway-76bbc4bf78-xmjpv
Namespace:        istio-system
Priority:         0
Service Account:  cluster-local-gateway-service-account
Node:             edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us/128.110.218.150
Start Time:       Tue, 05 Mar 2024 02:16:53 -0700
Labels:           app=cluster-local-gateway
                  chart=gateways
                  heritage=Tiller
                  install.operator.istio.io/owning-resource=unknown
                  istio=cluster-local-gateway
                  istio.io/rev=default
                  operator.istio.io/component=IngressGateways
                  pod-template-hash=76bbc4bf78
                  release=istio
                  service.istio.io/canonical-name=cluster-local-gateway
                  service.istio.io/canonical-revision=latest
                  sidecar.istio.io/inject=false
Annotations:      cni.projectcalico.org/containerID: 1b4b8a4991baa26046ca537fb6a765198453cb4e7848cc35cbec4d3a28044394
                  cni.projectcalico.org/podIP: 192.168.157.193/32
                  cni.projectcalico.org/podIPs: 192.168.157.193/32
                  prometheus.io/path: /stats/prometheus
                  prometheus.io/port: 15020
                  prometheus.io/scrape: true
                  sidecar.istio.io/inject: false
Status:           Running
IP:               192.168.157.193
IPs:
  IP:           192.168.157.193
Controlled By:  ReplicaSet/cluster-local-gateway-76bbc4bf78
Containers:
  istio-proxy:
    Container ID:  containerd://343cad521d50edf23bfa3d741a08034082ab1962663f354f1fdd7da84b2633a7
    Image:         docker.io/istio/proxyv2:1.16.3
    Image ID:      docker.io/istio/proxyv2@sha256:35ecc61d241242e8d68746fcccb253c4abc7d3b7671702ddb9e20b532cc514f2
    Ports:         15020/TCP, 8080/TCP, 8443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
    State:          Running
      Started:      Tue, 05 Mar 2024 02:16:59 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                   first-party-jwt
      PILOT_CERT_PROVIDER:          istiod
      CA_ADDR:                      istiod.istio-system.svc:15012
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     cluster-local-gateway-76bbc4bf78-xmjpv (v1:metadata.name)
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      SERVICE_ACCOUNT:               (v1:spec.serviceAccountName)
      ISTIO_META_WORKLOAD_NAME:     cluster-local-gateway
      ISTIO_META_OWNER:             kubernetes://apis/apps/v1/namespaces/istio-system/deployments/cluster-local-gateway
      ISTIO_META_MESH_ID:           cluster.local
      TRUST_DOMAIN:                 cluster.local
      ISTIO_META_UNPRIVILEGED_POD:  true
      ISTIO_META_CLUSTER_ID:        Kubernetes
    Mounts:
      /etc/istio/config from config-volume (rw)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /etc/istio/pod from podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/credential-uds from credential-socket (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzjnl (ro)
      /var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
      /var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  workload-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  credential-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  workload-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  kube-api-access-dzjnl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                      From               Message
  ----     ------     ----                     ----               -------
  Normal   Scheduled  9m21s                    default-scheduler  Successfully assigned istio-system/cluster-local-gateway-76bbc4bf78-xmjpv to edge-0.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us
  Normal   Pulling    9m21s                    kubelet            Pulling image "docker.io/istio/proxyv2:1.16.3"
  Normal   Pulled     9m15s                    kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.16.3" in 5.809399086s (5.809488756s including waiting)
  Normal   Created    9m15s                    kubelet            Created container istio-proxy
  Normal   Started    9m15s                    kubelet            Started container istio-proxy
  Warning  Unhealthy  4m19s (x154 over 9m14s)  kubelet            Readiness probe failed: Get "http://192.168.157.193:15021/healthz/ready": dial tcp 192.168.157.193:15021: connect: connection refused
yulinzou commented 7 months ago

kubectl describe pod istiod-657b54846b-2ncl8 -n istio-system

Name:             istiod-657b54846b-2ncl8
Namespace:        istio-system
Priority:         0
Service Account:  istiod
Node:             edge-1.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us/128.110.218.125
Start Time:       Tue, 05 Mar 2024 02:16:43 -0700
Labels:           app=istiod
                  install.operator.istio.io/owning-resource=unknown
                  istio=pilot
                  istio.io/rev=default
                  operator.istio.io/component=Pilot
                  pod-template-hash=657b54846b
                  sidecar.istio.io/inject=false
Annotations:      cni.projectcalico.org/containerID: 5cdc8c69e6be7075b384931191d7f08a742cc61928743a598e886165e56d6642
                  cni.projectcalico.org/podIP: 192.168.32.1/32
                  cni.projectcalico.org/podIPs: 192.168.32.1/32
                  prometheus.io/port: 15014
                  prometheus.io/scrape: true
                  sidecar.istio.io/inject: false
Status:           Running
IP:               192.168.32.1
IPs:
  IP:           192.168.32.1
Controlled By:  ReplicaSet/istiod-657b54846b
Containers:
  discovery:
    Container ID:  containerd://f1789e0fd9199390262c119cfca3a118ab2c7bb127298e07b9860a87fd958174
    Image:         docker.io/istio/pilot:1.16.3
    Image ID:      docker.io/istio/pilot@sha256:91a8907fee81051fe22d767cbca2584d1b07b475c686403395a7207d82e8f36e
    Ports:         8080/TCP, 15010/TCP, 15017/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      discovery
      --monitoringAddr=:15014
      --log_output_level=default:info
      --domain
      cluster.local
      --keepaliveMaxServerConnectionAge
      30m
    State:          Running
      Started:      Tue, 05 Mar 2024 02:16:48 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      500m
      memory:   2Gi
    Readiness:  http-get http://:8080/ready delay=1s timeout=5s period=3s #success=1 #failure=3
    Environment:
      REVISION:                                     default
      JWT_POLICY:                                   first-party-jwt
      PILOT_CERT_PROVIDER:                          istiod
      POD_NAME:                                     istiod-657b54846b-2ncl8 (v1:metadata.name)
      POD_NAMESPACE:                                istio-system (v1:metadata.namespace)
      SERVICE_ACCOUNT:                               (v1:spec.serviceAccountName)
      KUBECONFIG:                                   /var/run/secrets/remote/config
      PILOT_TRACE_SAMPLING:                         1
      PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_OUTBOUND:  true
      PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_INBOUND:   true
      ISTIOD_ADDR:                                  istiod.istio-system.svc:15012
      PILOT_ENABLE_ANALYSIS:                        false
      CLUSTER_ID:                                   Kubernetes
    Mounts:
      /etc/cacerts from cacerts (ro)
      /var/run/secrets/istio-dns from local-certs (rw)
      /var/run/secrets/istiod/ca from istio-csr-ca-configmap (ro)
      /var/run/secrets/istiod/tls from istio-csr-dns-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rvxxs (ro)
      /var/run/secrets/remote from istio-kubeconfig (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  local-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  cacerts:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cacerts
    Optional:    true
  istio-kubeconfig:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-kubeconfig
    Optional:    true
  istio-csr-dns-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istiod-tls
    Optional:    true
  istio-csr-ca-configmap:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  true
  kube-api-access-rvxxs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  34m                default-scheduler  Successfully assigned istio-system/istiod-657b54846b-2ncl8 to edge-1.yulin001-194981.ntu-cloud-pg0.utah.cloudlab.us
  Normal   Pulling    34m                kubelet            Pulling image "docker.io/istio/pilot:1.16.3"
  Normal   Pulled     34m                kubelet            Successfully pulled image "docker.io/istio/pilot:1.16.3" in 4.771692006s (4.771701671s including waiting)
  Normal   Created    34m                kubelet            Created container discovery
  Normal   Started    34m                kubelet            Started container discovery
  Warning  Unhealthy  34m (x2 over 34m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503