search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.21k stars 316 forks source link

istio-ingressgateway is not getting a load balancer External IP load balancer end-point(AWS eks) #597

Open abhinavkorpal opened 4 years ago

abhinavkorpal commented 4 years ago

Tell us about your request What do you want us to build? istio

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                      AGE                                                                                  2m47s
service/istio-ingressgateway     LoadBalancer   x.x.x.x    <pending>     x.x.x.x.x.x   2m48s

Are you currently working around this issue? ingress-gateway

Additional context

Warning  Unhealthy  110s (x3 over 114s)  kubelet, ip-x-x-x-x.ec2.internal  Readiness probe failed: HTTP probe failed with statuscode: 503

**Attachments**
```shell
LAST SEEN   TYPE      REASON                         OBJECT                                         MESSAGE
2m7s        Normal    Scheduled                      pod/istio-citadel-56465d79b9-czpgg             Successfully assigned istio-system/istio-citadel-56465d79b9-czpgg to ip-x-x-x-x.ec2.internal
2m6s        Normal    Pulling                        pod/istio-citadel-56465d79b9-czpgg             Pulling image "docker.io/istio/citadel:1.4.0"
2m5s        Normal    Pulled                         pod/istio-citadel-56465d79b9-czpgg             Successfully pulled image "docker.io/istio/citadel:1.4.0"
2m4s        Normal    Created                        pod/istio-citadel-56465d79b9-czpgg             Created container citadel
2m4s        Normal    Started                        pod/istio-citadel-56465d79b9-czpgg             Started container citadel
2m7s        Normal    SuccessfulCreate               replicaset/istio-citadel-56465d79b9            Created pod: istio-citadel-56465d79b9-czpgg
2m7s        Normal    ScalingReplicaSet              deployment/istio-citadel                       Scaled up replica set istio-citadel-56465d79b9 to 1
2m4s        Normal    Scheduled                      pod/istio-galley-6b9495645d-hmpbd              Successfully assigned istio-system/istio-galley-6b9495645d-hmpbd to ip-x-x-x-x.ec2.internal
116s        Warning   FailedMount                    pod/istio-galley-6b9495645d-hmpbd              MountVolume.SetUp failed for volume "istio-certs" : secret "istio.istio-galley-service-account" not found
107s        Normal    Pulled                         pod/istio-galley-6b9495645d-hmpbd              Container image "docker.io/istio/galley:1.4.0" already present on machine
107s        Normal    Created                        pod/istio-galley-6b9495645d-hmpbd              Created container galley
107s        Normal    Started                        pod/istio-galley-6b9495645d-hmpbd              Started container galley
107s        Normal    Pulled                         pod/istio-galley-6b9495645d-hmpbd              Container image "docker.io/istio/proxyv2:1.4.0" already present on machine
107s        Normal    Created                        pod/istio-galley-6b9495645d-hmpbd              Created container istio-proxy
107s        Normal    Started                        pod/istio-galley-6b9495645d-hmpbd              Started container istio-proxy
2m4s        Normal    SuccessfulCreate               replicaset/istio-galley-6b9495645d             Created pod: istio-galley-6b9495645d-hmpbd
2m4s        Normal    ScalingReplicaSet              deployment/istio-galley                        Scaled up replica set istio-galley-6b9495645d to 1
56s         Normal    DeletingLoadBalancer           service/istio-ingress                          Deleting load balancer
2m8s        Normal    Scheduled                      pod/istio-ingressgateway-5df799fdbd-b7spf      Successfully assigned istio-system/istio-ingressgateway-5df799fdbd-b7spf to ip-x-x-x-x.ec2.internal
2m7s        Normal    Pulled                         pod/istio-ingressgateway-5df799fdbd-b7spf      Container image "docker.io/istio/proxyv2:1.4.0" already present on machine
2m7s        Normal    Created                        pod/istio-ingressgateway-5df799fdbd-b7spf      Created container istio-proxy
2m7s        Normal    Started                        pod/istio-ingressgateway-5df799fdbd-b7spf      Started container istio-proxy
82s         Warning   Unhealthy                      pod/istio-ingressgateway-5df799fdbd-b7spf      Readiness probe failed: Get http://x.x.x.x:x/healthz/ready: dial tcp x.x.x.x:x: connect: connection refused
2m8s        Normal    SuccessfulCreate               replicaset/istio-ingressgateway-5df799fdbd     Created pod: istio-ingressgateway-5df799fdbd-b7spf
2m3s        Normal    DeletingLoadBalancer           service/istio-ingressgateway                   Deleting load balancer
2m8s        Normal    ScalingReplicaSet              deployment/istio-ingressgateway                Scaled up replica set istio-ingressgateway-5df799fdbd to 1
1s          Warning   FailedGetResourceMetric        horizontalpodautoscaler/istio-ingressgateway   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
1s          Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/istio-ingressgateway   failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
2m4s        Normal    Scheduled                      pod/istio-pilot-c9bdc55bd-bh2dl                Successfully assigned istio-system/istio-pilot-c9bdc55bd-bh2dl to ip-x-x-x-x.ec2.internal
2m3s        Normal    Pulled                         pod/istio-pilot-c9bdc55bd-bh2dl                Container image "docker.io/istio/pilot:1.4.0" already present on machine
2m3s        Normal    Created                        pod/istio-pilot-c9bdc55bd-bh2dl                Created container discovery
2m3s        Normal    Started                        pod/istio-pilot-c9bdc55bd-bh2dl                Started container discovery
2m3s        Normal    Pulled                         pod/istio-pilot-c9bdc55bd-bh2dl                Container image "docker.io/istio/proxyv2:1.4.0" already present on machine
2m3s        Normal    Created                        pod/istio-pilot-c9bdc55bd-bh2dl                Created container istio-proxy
2m3s        Normal    Started                        pod/istio-pilot-c9bdc55bd-bh2dl                Started container istio-proxy
16s         Warning   Unhealthy                      pod/istio-pilot-c9bdc55bd-bh2dl                Readiness probe failed: HTTP probe failed with statuscode: 503
2m4s        Normal    SuccessfulCreate               replicaset/istio-pilot-c9bdc55bd               Created pod: istio-pilot-c9bdc55bd-bh2dl
2m4s        Normal    ScalingReplicaSet              deployment/istio-pilot                         Scaled up replica set istio-pilot-c9bdc55bd to 1
1s          Warning   FailedGetResourceMetric        horizontalpodautoscaler/istio-pilot            unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
1s          Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/istio-pilot            failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
2m7s        Normal    Scheduled                      pod/istio-policy-74cf89cb66-4thgn              Successfully assigned istio-system/istio-policy-74cf89cb66-4thgn to ip-x-x-x-x.ec2.internal
2m6s        Normal    Pulling                        pod/istio-policy-74cf89cb66-4thgn              Pulling image "docker.io/istio/mixer:1.4.0"
2m4s        Normal    Pulled                         pod/istio-policy-74cf89cb66-4thgn              Successfully pulled image "docker.io/istio/mixer:1.4.0"
106s        Normal    Created                        pod/istio-policy-74cf89cb66-4thgn              Created container mixer
106s        Normal    Started                        pod/istio-policy-74cf89cb66-4thgn              Started container mixer
2m4s        Normal    Pulled                         pod/istio-policy-74cf89cb66-4thgn              Container image "docker.io/istio/proxyv2:1.4.0" already present on machine
2m4s        Normal    Created                        pod/istio-policy-74cf89cb66-4thgn              Created container istio-proxy
2m4s        Normal    Started                        pod/istio-policy-74cf89cb66-4thgn              Started container istio-proxy
107s        Warning   Unhealthy                      pod/istio-policy-74cf89cb66-4thgn              Liveness probe failed: Get http://x.x.x.x:x/healthz/ready: dial tcp x.x.x.x:x: connect: connection refused
107s        Normal    Killing                        pod/istio-policy-74cf89cb66-4thgn              Container mixer failed liveness probe, will be restarted
106s        Normal    Pulled                         pod/istio-policy-74cf89cb66-4thgn              Container image "docker.io/istio/mixer:1.4.0" already present on machine
2m7s        Normal    SuccessfulCreate               replicaset/istio-policy-74cf89cb66             Created pod: istio-policy-74cf89cb66-4thgn
2m7s        Normal    ScalingReplicaSet              deployment/istio-policy                        Scaled up replica set istio-policy-74cf89cb66 to 1
28s         Normal    NoPods                         poddisruptionbudget/istio-policy               No matching pods found
1s          Warning   FailedGetResourceMetric        horizontalpodautoscaler/istio-policy           unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
1s          Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/istio-policy           failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
2m3s        Normal    Scheduled                      pod/istio-sidecar-injector-759bf6b4bc-2tt5t    Successfully assigned istio-system/istio-sidecar-injector-759bf6b4bc-2tt5t to ip-x-x-x-x.ec2.internal
2m1s        Warning   FailedMount                    pod/istio-sidecar-injector-759bf6b4bc-2tt5t    MountVolume.SetUp failed for volume "certs" : secret "istio.istio-sidecar-injector-service-account" not found
119s        Normal    Pulling                        pod/istio-sidecar-injector-759bf6b4bc-2tt5t    Pulling image "docker.io/istio/sidecar_injector:1.4.0"
113s        Normal    Pulled                         pod/istio-sidecar-injector-759bf6b4bc-2tt5t    Successfully pulled image "docker.io/istio/sidecar_injector:1.4.0"
113s        Normal    Created                        pod/istio-sidecar-injector-759bf6b4bc-2tt5t    Created container sidecar-injector-webhook
112s        Normal    Started                        pod/istio-sidecar-injector-759bf6b4bc-2tt5t    Started container sidecar-injector-webhook
2m3s        Normal    SuccessfulCreate               replicaset/istio-sidecar-injector-759bf6b4bc   Created pod: istio-sidecar-injector-759bf6b4bc-2tt5t
2m3s        Normal    ScalingReplicaSet              deployment/istio-sidecar-injector              Scaled up replica set istio-sidecar-injector-759bf6b4bc to 1
28s         Normal    NoPods                         poddisruptionbudget/istio-sidecar-injector     No matching pods found
2m5s        Normal    Scheduled                      pod/istio-telemetry-77b6dfb4ff-cv8wj           Successfully assigned istio-system/istio-telemetry-77b6dfb4ff-cv8wj to ip-x-x-x-x.ec2.internal
106s        Normal    Pulled                         pod/istio-telemetry-77b6dfb4ff-cv8wj           Container image "docker.io/istio/mixer:1.4.0" already present on machine
106s        Normal    Created                        pod/istio-telemetry-77b6dfb4ff-cv8wj           Created container mixer
106s        Normal    Started                        pod/istio-telemetry-77b6dfb4ff-cv8wj           Started container mixer
2m4s        Normal    Pulled                         pod/istio-telemetry-77b6dfb4ff-cv8wj           Container image "docker.io/istio/proxyv2:1.4.0" already present on machine
2m4s        Normal    Created                        pod/istio-telemetry-77b6dfb4ff-cv8wj           Created container istio-proxy
2m4s        Normal    Started                        pod/istio-telemetry-77b6dfb4ff-cv8wj           Started container istio-proxy
101s        Warning   Unhealthy                      pod/istio-telemetry-77b6dfb4ff-cv8wj           Liveness probe failed: Get http://x.x.x.x:x/healthz/ready: dial tcp x.x.x.x:x: connect: connection refused
106s        Normal    Killing                        pod/istio-telemetry-77b6dfb4ff-cv8wj           Container mixer failed liveness probe, will be restarted
2m5s        Normal    SuccessfulCreate               replicaset/istio-telemetry-77b6dfb4ff          Created pod: istio-telemetry-77b6dfb4ff-cv8wj
2m5s        Normal    ScalingReplicaSet              deployment/istio-telemetry                     Scaled up replica set istio-telemetry-77b6dfb4ff to 1
28s         Normal    NoPods                         poddisruptionbudget/istio-telemetry            No matching pods found
13s         Warning   FailedGetResourceMetric        horizontalpodautoscaler/istio-telemetry        unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
13s         Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/istio-telemetry        failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
2m5s        Normal    Scheduled                      pod/prometheus-586d4445c7-cvhxr                Successfully assigned istio-system/prometheus-586d4445c7-cvhxr to ip-x-x-x-x.ec2.internal
118s        Warning   FailedMount                    pod/prometheus-586d4445c7-cvhxr                MountVolume.SetUp failed for volume "istio-certs" : secret "istio.default" not found
109s        Normal    Pulling                        pod/prometheus-586d4445c7-cvhxr                Pulling image "docker.io/prom/prometheus:v2.12.0"
105s        Normal    Pulled                         pod/prometheus-586d4445c7-cvhxr                Successfully pulled image "docker.io/prom/prometheus:v2.12.0"
105s        Normal    Created                        pod/prometheus-586d4445c7-cvhxr                Created container prometheus
105s        Normal    Started                        pod/prometheus-586d4445c7-cvhxr                Started container prometheus
2m5s        Normal    SuccessfulCreate               replicaset/prometheus-586d4445c7               Created pod: prometheus-586d4445c7-cvhxr
2m5s        Normal    ScalingReplicaSet              deployment/prometheus                          Scaled up replica set prometheus-586d4445c7 to 1
tabern commented 4 years ago

@abhinavkorpal this seems like a technical support issue. Have you contacted AWS support about this?