search

seriohub / velero-helm

Helm charts for velero-ui, velero-api, and velero-watchdog
6 stars 3 forks source link

When attempting to upgrade helm from 0.1.7 to 0.1.8, api pod is trying to spin up and attach the PVC that is still mounted #20

Closed nickveldrin closed 3 weeks ago

nickveldrin commented 1 month ago

I use terraform for my helm and kubernetes resources, so I adjusted the version from 0.1.7 to 0.1.8 and saw that the API pod was stuck trying to create.

When i described the pod, i got this error:

 Warning  FailedAttachVolume  44s   attachdetach-controller  Multi-Attach error for volume "pvc-447e3615-0fce-4d81-b48d-64e49ecc95b9" Volume is already used by pod(s) velero-ui-vui-api-859cb75f69-6rkhz

kubectl get pod for the namespace looks like this:

kubectl -n velero-ui get pod
NAME                                     READY   STATUS              RESTARTS   AGE
velero-ui-vui-api-547d97b6bc-mfdgf       0/1     ContainerCreating   0          4m47s
velero-ui-vui-api-859cb75f69-6rkhz       1/1     Running             0          2d1h
velero-ui-vui-ui-694cb66c78-5shh5        1/1     Running             0          4m47s
velero-ui-vui-watchdog-bcbbc8dfd-rl7b7   1/1     Running             0          4m47s

So the helm upgrade is stuck.

I'm not well versed enough in the deployment spec or helm options, but i would assume there's a way to handle this since i've seen other applications with pvc's that are able to terminate.

davideserio commented 1 month ago

Hi, I don't know why this happened; we have made several upgrades using Helm without any problems. Right now, I don't have the ability to replicate this issue using Terraform, but I can investigate similar cases or options that might help. In the meantime, if you haven't resolved it yet, you could try manually stopping the API pod or reinstalling from scratch.

nickveldrin commented 1 month ago

I just deleted the old api pod and the new one came up, so it's okay there.

But yeah, I'm not exactly sure if it's my issue or a parameter/setting that can be adjusted in the chart/chart helper/deployment somewhere.

Feel free to close unless other folks report issues as well.

davideserio commented 1 month ago

Thank you for the report. I will leave it open for a while in case there are further reports.

nickveldrin commented 1 month ago

It looks like the parameter you need to set is in the deployment spec: StrategyType should be "Recreate".

By default it is "RollingUpdate", which won't work for pods with PVC's.

oliverbaehler commented 1 month ago

@nickveldrin unless your PVC is RWM.

nickveldrin commented 1 month ago

@nickveldrin unless your PVC is RWM.

Yep!