search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

Not deploying to EKS #702

Open mckunkel opened 2 years ago

mckunkel commented 2 years ago

Describe the bug

Using EKS templates provided in the fix-eks-issue branch. Using the main branch to deploy cf-for-k8.

kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)

Error 12:10:19PM: fail: reconcile deployment/eirini-api (apps/v1) namespace: cf-system 12:10:19PM: ^ Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-api-85c488b6f5" has timed out progressing.)

kapp: Error: waiting on reconcile deployment/eirini-api (apps/v1) namespace: cf-system: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-api-85c488b6f5" has timed out progressing.))

Tried again 12:09:21PM: fail: reconcile deployment/log-cache-backend (apps/v1) namespace: cf-system 12:09:21PM: ^ Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "log-cache-backend-c5675b68b" has timed out progressing.)

kapp: Error: waiting on reconcile deployment/log-cache-backend (apps/v1) namespace: cf-system: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "log-cache-backend-c5675b68b" has timed out progressing.))

Third attempt: 12:11:06PM: ^ Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-instance-index-env-injector-6b7f479874" has timed out progressing.)

kapp: Error: waiting on reconcile deployment/eirini-instance-index-env-injector (apps/v1) namespace: cf-system: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-instance-index-env-injector-6b7f479874" has timed out progressing.))

Everytime a different failure.

To Reproduce*

Steps to reproduce the behavior:

  1. go to deploy/eks/terraform and create eks cluster. Tested with versions 1.19, 1.20, and 1.21
  2. Create template file as prescribed in walkthrough, mine ended as `app_registry: hostname: https://index.docker.io/v1/ repository_prefix: "mckunkel" username: "mckunkel" password: "XXX"

add_metrics_server_components: true enable_automount_service_account_token: true load_balancer: enable: false metrics_server_prefer_internal_kubelet_address: true remove_resource_requirements: true use_first_party_jwt_tokens: true`

  1. Use kapp to deploy kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)
  2. See error

It's helpful to include snippets of the error response or logs output

Expected behavior

A clear and concise description of what you expected to happen. cf-for-k8 to deploy to eks sucessfully

Additional context

cf-for-k8s SHA

Please paste cf-for-k8s SHA hyperlink

Deploy instructions

Please include the kapp deploy... command, including all config-optional files kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)

Cluster information

Please provide IaaS name(s) here (GKE, AKS, EKS, minikube, kind, etc.) EKS

CLI versions

paste output of the following commands

  1. ytt --version: 0.40.0
  2. kapp --version:0 .46.0
  3. kubectl version: 1.21, 1.20, 1.19
  4. cf version: 7.4.0+e55633fed.2021-11-15
cf-gitbot commented 2 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/181856385

The labels on this github issue will be updated when the story is started.

mckunkel commented 2 years ago

Further into the investigation it seems that all failures point to isitio. This is output that all kubectl describe ReplicaSet/\<from replica list> -n \<replica namespace>

`
Type Reason Age From Message

Warning FailedCreate 15m (x8 over 15m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": service "istiod" not found Warning FailedCreate 15m (x4 over 15m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": no endpoints available for service "istiod" Warning FailedCreate 8m41s replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.0.3.190:15017: i/o timeout Warning FailedCreate 3m4s (x5 over 14m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded `

I do not see any resolutions in the issues page on this. Can I be guided in the correct path to fix this please?