Cannot bind volume on EKS

alexnimo commented 1 year ago

What happened:

Tried to install apiclarity on EKS version 1.24. Tried it on managed node group and on Fargate. The ebs-csi controller is provisioned. Both of the apiclarity pods are stuck on pending and return the following error: Warning FailedScheduling 9m33s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

What you expected to happen:

Pods to start normally

How to reproduce it (as minimally and precisely as possible):

EKS cluster version 1.24

Are there any error messages in API Clarity logs?

Warning FailedScheduling 9m33s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

Anything else we need to know?:

Environment:

Kubernetes version 1.24
Istio version (use istioctl version): client version: 1.16.1 control plane version: 1.11.4 data plane version: 1.11.4 (1 proxies)
APIClarity version (use kubectl -n apiclarity exec deploy/apiclarity -- ./backend version)
Cloud provider or hardware configuration:
AWS + EKS
Others:

akpsgit commented 1 year ago

Hi @alexnimo, can this be related to https://stackoverflow.com/questions/68725070/k8s-pods-failure-error-while-running-volumebinding-prebind-plugin-for-pod-a?

alexnimo commented 1 year ago

Hi @akpsgit , Nope, it looks like something else. The error message on the PVC is: Normal Provisioning 3m17s (x688 over 42h) ebs.csi.aws.com_ebs-csi-controller-7bffdbb6d6-rt8td_0f667e1c-20f9-49ab-8ae9-db2b72131365 External provisioner is provisioning volume for claim "apiclarity/data-apiclarity-apiclarity-postgresql-0" Normal ExternalProvisioning 2m37s (x10264 over 42h) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator OIDC was already in use and working with different pods so not sure it's related....

pkalapat commented 1 year ago

To resolve this issue it needs Amazon EBS CSI driver as an Amazon EKS add-on and setting up the driver with relevant IAMServiceAccount roles and policies. Post eks1.23 it is required to install add ons. Some useful references

https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html

First uninistall your current installation of API Clarity with helm helm uninstall apiclarity -n apiclarity

Next, follow the instructions provided in the above links to install CSI add ons in your cluster and give it IAMServiceRole with associated trust policy

Now re-try apiclarity install

If this does not work you may have to delete your existing cluster, configure with add on and then attempt apiclarity install.

openclarity / apiclarity