search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

istio cert not installing #624

Closed dlhace closed 3 years ago

dlhace commented 3 years ago

Describe the bug

I am following the documentation for installing cf-for-k8s into a rancher created kubernetes cluster with the ingress disabled.

When I deploy, I am getting the error "kapp: Error: waiting on reconcile deployment/eirini-controller (apps/v1) namespace: cf-system: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "eirini-controller-7b5ccdcc58" has timed out progressing.))"

The root cause appears to be that it is waiting for the completion of istio and istio cannot find the certificates.

Warning FailedMount 52m kubelet, h2-worker1 Unable to attach or mount volumes: unmounted volumes=[istio-token istiod-ca-cert], unattached volumes=[ingressgateway-certs istio-envoy istio-ingressgateway-service-account-token-hpdms varlog fluent-bit-config gatewaysdsudspath istio-token podinfo ingressgateway-ca-certs config-volume dockercontainers istiod-ca-cert]: timed out waiting for the condition Warning FailedMount 6m47s (x26 over 49m) kubelet, h2-worker1 (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert istio-token], unattached volumes=[istio-envoy gatewaysdsudspath fluent-bit-config istiod-ca-cert ingressgateway-certs podinfo ingressgateway-ca-certs varlog istio-token istio-ingressgateway-service-account-token-hpdms dockercontainers config-volume]: timed out waiting for the condition Warning FailedMount 2m29s (x34 over 67m) kubelet, h2-worker1 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

Did I miss a step?

To Reproduce*

Steps to reproduce the behavior:

  1. Created a rancher vsphere based kubernetes cluster with no ingress controller installed

  2. git clone https://github.com/cloudfoundry/cf-for-k8s.git -b main

  3. cd cf-for-k8s

  4. TMP_DIR=../tmp; mkdir -p ${TMP_DIR}

  5. ./hack/generate-values.sh -d cf-k8.example.com > ${TMP_DIR}/cf-values.yml

  6. cat << EOF >> ${TMP_DIR}/cf-values.yml app_registry: hostname: https://xxx.azure.io repository_prefix: "xxxx" username: "xxxx" password: "secret" EOF

  7. ytt -f config -f ${TMP_DIR}/cf-values.yml > ${TMP_DIR}/cf-for-k8s-rendered.yml

  8. kapp deploy -a cf -f ${TMP_DIR}/cf-for-k8s-rendered.yml -y

It's helpful to include snippets of the error response or logs output

Expected behavior

cf for k9s to install

cf-for-k8s SHA

Please paste cf-for-k8s SHA hyperlink

Deploy instructions

see to "Reproduce"

Cluster information

rks (rancher kubernetes)

NAME STATUS ROLES AGE VERSION OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME h2-master1 Ready controlplane,etcd 41h v1.19.7 Ubuntu 20.04.2 LTS 5.4.0-65-generic docker://19.3.15 h2-worker1 Ready worker 41h v1.19.7 Ubuntu 20.04.2 LTS 5.4.0-65-generic docker://19.3.15 h2-worker2 Ready worker 41h v1.19.7 Ubuntu 20.04.2 LTS 5.4.0-65-generic docker://19.3.15 h2-worker3 Ready worker 41h v1.19.7 Ubuntu 20.04.2 LTS 5.4.0-65-generic docker://19.3.15 h2-worker4 Ready worker 41h v1.19.7 Ubuntu 20.04.2 LTS 5.4.0-65-generic docker://19.3.15 h2-worker5 Ready worker 41h v1.19.7 Ubuntu 20.04.2 LTS 5.4.0-65-generic docker://19.3.15

CLI versions

paste output of the following commands

  1. ytt --version: 0.31.0
  2. kapp --version: 0.35.0
  3. kubectl version: v1.20.2
  4. cf version: 7.2.0+be4a5ce2b.2020-12-10
cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/176952333

The labels on this github issue will be updated when the story is started.

dlhace commented 3 years ago

Also I should note that for the loadbalancer, I use metallb and the nodes are all 8 cpu x 16 GB x 100 GB disk space and I have tried both nfs and vsphere provisioners. All of these components I know work, because I used them on other kubernetes clusters like kubecf and personal websites and containers.

acosta11 commented 3 years ago

Hi @dlhace,

Thanks for the detailed reproduction and error information! I think you may be running into the same issue as https://github.com/cloudfoundry/cf-for-k8s/issues/542 where the enable_automount_service_account_token and use_first_party_jwt_tokens configuration options are required on the vsphere environment due to the absence of an integrated/pre-configured jwt provider. If you set each of those configuration values to true in your cf-values.yml file, does istio successfully mount its volumes and eventually reconcile on Eirini?

Thanks, Andrew and @Birdrock

dlhace commented 3 years ago

Thanks Andrew, Adding those entries allowed the istio containers to load. Now on to the next problem

On Tue, Feb 16, 2021 at 3:19 PM Andrew Costa notifications@github.com wrote:

Hi @dlhace https://github.com/dlhace,

Thanks for the detailed reproduction and error information! I think you may be running into the same issue as #542 https://github.com/cloudfoundry/cf-for-k8s/issues/542 where the enable_automount_service_account_tokens and use_first_party_jwt configuration options are required on the vsphere environment due to the absence of an integrated/pre-configured jwt provider. If you set each of those configuration values to true in your cf-values.yml file, does istio successfully mount its volumes and eventually reconcile on Eirini?

Thanks, Andrew and @Birdrock https://github.com/Birdrock

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cloudfoundry/cf-for-k8s/issues/624#issuecomment-780154386, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRQM2XIEOWU7UOJPO5XUF3S7LVOVANCNFSM4XVLMO3Q .