search

vultr / vultr-csi

Container Storage Interface (CSI) Driver for Vultr Block Storage
Apache License 2.0
40 stars 17 forks source link

Unable to delete PVs #68

Closed robcxyz closed 2 years ago

robcxyz commented 2 years ago

I have a volume that is unable to be detached with the following error:

time="2022-04-04T21:24:18Z" level=error msg="GRPC error: rpc error: code = Internal desc = cannot detach volume: {\"error\":\"Unable to detach block storage\",\"status\":400}" GRPC.call=/csi.v1.Controller/ControllerUnpublishVolume GRPC.request="volume_id:\"59cda749-b10d-4372-970d-6c91d3c52cd9\" node_id:\"ba14e0d5-df7f-4e31-94c5-78bb69778644\" "
time="2022-04-04T21:24:18Z" level=error msg="GRPC error: rpc error: code = Internal desc = cannot detach volume: {\"error\":\"Unable to detach block storage\",\"status\":400}" GRPC.call=/csi.v1.Controller/ControllerUnpublishVolume GRPC.request="volume_id:\"59cda749-b10d-4372-970d-6c91d3c52cd9\" node_id:\"ba14e0d5-df7f-4e31-94c5-78bb69778644\" "

Pod has been stopped and the container is stuck in Status: Terminating. Trying to delete it and I get the log messages above. I am generally able to release volumes but a few are stuck. Also tried deleting the volume in the UI but unable to detach it from the instance.

Can't really find a pattern with this so I am sorry if I am not giving all the information needed. Will also file a ticket with support and happy to follow up with more info.

ddymko commented 2 years ago

@robcxyz The support ticket will end up coming to me.

This doesn't really seem to be a CSI issue as much as it is a API issue {\"error\":\"Unable to detach block storage\",\"status\":400} this is the response that the CSI is getting from the API

Will look into

ddymko commented 2 years ago

I just tried to replicate this on a VKE cluster running in EWR...attaching and detaching worked ok for me.

One thing to note

Also tried deleting the volume in the UI but unable to detach it from the instance.

If you are unable to detach the instance from the UI will be an issue with the specific block storage itself. This would explain why the CSI/API are having issues detaching.

I raised this issue with members of the storage team so they can take a look as to why the detach (on UI and API) aren't working