Cloudstack volume is not deleted when PVC is deleted

ISSUE TYPE

Bug Report

COMPONENT NAME

Storage volumes, Kubernetes Service

CLOUDSTACK VERSION

4.17.2.0

CONFIGURATION

Advanced networking with vxlan isolation of guest networks
Kubernetes service enabled
Cloudstack CSI driver deployed

OS / ENVIRONMENT

Rocky Linnux 8.7

SUMMARY

When a Kubernetes persistent volume claim is deleted, the Cloudstack volume is not deleted, and an error message is written to the management log.

STEPS TO REPRODUCE

Install the CSI driver in the K8s cluster using the command :

kubectl apply -f https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml

Deploy this storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cloudstack-custom
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
csi.cloudstack.apache.org/disk-offering-id: <id of custom disk offering>

Deploy the following pvc and pod:


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: example-pvc
spec:
storageClassName: cloudstack-custom
accessModes:
- ReadWriteOnce
resources:
requests:
  storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example
  image: busybox
  volumeMounts:
    - mountPath: "/data"
      name: example-volume
  stdin: true
  stdinOnce: true
  tty: true
volumes:
- name: example-volume
  persistentVolumeClaim:
    claimName: example-pvc

* Delete the pod and pvc

##### EXPECTED RESULTS

I expected the Cloudstack storage volume backing the PVC would be deleted.


##### ACTUAL RESULTS
The storage volume was not deleted, and the following error was printed to the management log:

2023-03-05 14:18:39,144 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658 ctx-d6957fbc) (logid:fd682bcf) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment 2023-03-05 14:18:39,144 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658 ctx-d6957fbc) (logid:fd682bcf) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment 2023-03-05 14:18:39,144 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658) (logid:fd682bcf) Done with run of VM work job: com.cloud.vm.VmWorkDetachVolume for VM 188, job origin: 1657 2023-03-05 14:18:39,144 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658) (logid:fd682bcf) Unable to complete AsyncJobVO: {id:1658, userId: 4, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkDetachVolume, cmdInfo: rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrRGV0YWNoVm9sdW1lhvReA-s81AMCAAFMAAh2b2x1bWVJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAAEAAAAAAAAALx0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAvA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 128437426857854, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sun Mar 05 14:18:36 EST 2023, removed: null}, job origin:1657 com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:2793) at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:4404) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4435) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at com.sun.proxy.$Proxy231.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

apalia / cloudstack-csi-driver