search

kubernetes-sigs / vsphere-csi-driver

vSphere storage Container Storage Interface (CSI) plugin
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/index.html
Apache License 2.0
295 stars 177 forks source link

disk doesn't deleted from VM #223

Closed gricuk closed 3 years ago

gricuk commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: Virtual disks which were created by csi driver don't detach from vm and not deleted when pvc volume was deleted. What you expected to happen: When I delete PV I expected that virtual disk will be deleted and detached from VM How to reproduce it (as minimally and precisely as possible): Delete pvc and check: fdisk -l

Disk /dev/sdh: 700 GiB, 751619276800 bytes, 1468006400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdd: 700 GiB, 751619276800 bytes, 1468006400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdg: 23.3 GiB, 25000148992 bytes, 48828416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdi: 23.3 GiB, 25000148992 bytes, 48828416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdf: 23.3 GiB, 25000148992 bytes, 48828416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdm: 23.3 GiB, 25000148992 bytes, 48828416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdl: 23.3 GiB, 25000148992 bytes, 48828416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdj: 700 GiB, 751619276800 bytes, 1468006400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdk: 232.9 GiB, 250000441344 bytes, 488282112 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Half of these disks should be deleted and detached from vm because I've deleted PV.

disk.ls -dc  FSN1 -ds vsanDatastore -l
08b841e8-d2cb-4535-af77-f57ab022cea6  pvc-67e22821-2d6a-4b51-aa72-eaee8eeb5f50  23.3G   May 13 11:31:38
0c1ff28f-1799-4673-b309-e07175a8c6a5  pvc-0c9b4270-0b92-41bf-b922-cc657ce188b4  1.9G    May  8 14:12:23
16808587-089a-4d7c-b4c6-30a1e4c2be3f  pvc-26446a01-b29d-40f5-a7cd-09c9fd0e9f38  700.0G  May 14 15:08:12
183490e7-1f66-4570-bcd1-9687e19c86d8  pvc-99dd1ead-6d65-4098-8507-27e5fe693e34  232.8G  May 14 15:08:12
1ec12684-523b-4f1c-800b-6293eef862db  pvc-cb81c7bc-0c55-41d4-ba5b-3198367b7a1c  23.3G   May 14 15:27:15
3a3c3739-0fb8-40d4-a42d-811eecf7f956  pvc-adcf2726-10d6-432a-8f7d-b2d37a9f6407  23.3G   May 11 14:03:38
3da39761-2d95-4dbc-b02a-20816bb05db6  pvc-09fb4f2c-0c3e-4bfc-8dbb-f5c99b5de125  93.1G   May  9 09:43:46
4dc72e6c-5c8b-4c2f-ae30-acb4649045c9  pvc-a2ed1b75-8cad-4162-ab25-69effdc95318  23.3G   May 14 11:50:04
4faa935e-caa6-44fe-9103-f9ba5c8f2785  pvc-fdcb4a37-ca27-4d71-8a4f-798ec48bdc02  700.0G  May 12 13:32:37
655e1917-2c91-4ed1-8020-aad285dc033c  pvc-697ac284-fb3e-48a2-a6f0-7af09501aa08  23.3G   May 14 15:08:12
7f09d66e-f2c8-4a51-b5ec-427efcd8b3fc  pvc-b96bf203-d3fb-45d7-b4d5-6ef049b5fca8  700.0G  May  9 09:24:26
95299472-d181-4db2-a578-62207bdfe512  pvc-0d3dbb40-b40d-4232-bfac-4060877036d2  167.6G  Apr 22 15:48:56
b3743779-afa2-427b-900e-1f6c5f079e0f  pvc-a04dcb13-3cf4-46ae-9fcd-64f6342eafc2  23.3G   May  9 21:24:14
b3868c12-069d-428f-94dc-f21549b8ea02  pvc-ed0a1d5d-075d-4ca3-a4c1-4dac3ea27b8f  23.3G   May 14 11:50:04
b3c1ddc8-99dc-4479-969f-a0b96cb9526f  pvc-c6735c3b-e412-40af-85db-b5e9b62a866f  23.3G   May 15 13:42:57
cb25b747-b94b-45ed-8760-1c1dabf904b0  pvc-bef9d05e-aed6-45bf-8aa8-a524dd163bd0  93.1G   May  9 09:24:26
e04c4680-1fd2-4d08-a8f1-cdd2b93db4ad  pvc-8c3ae9a2-3ed0-4be3-b6c6-06b9b7441a6c  167.6G  Apr 16 12:43:27
e06e722e-de34-466d-a12e-b83495b729e9  pvc-47c922cd-a051-4506-99e1-23a4f0397077  700.0G  May  9 09:43:50
e8c83b45-6093-4085-805f-f8c11d8268af  pvc-1f22be7e-e729-476d-833b-200cfa432ef1  700.0G  May 15 13:42:56
ec76c697-c0a5-4e69-a879-7c9229653b20  pvc-bb75da0f-a006-4ed4-834f-c83f32abb76d  1.9G    May  8 14:12:16

I'have found the following logs on vsphere-csi-controller:

"level":"error","time":"2020-05-09T10:14:42.239777303Z","caller":"vanilla/controller.go:562","msg":"volumeID \"cb25b747-b94b-45ed-8760-1c1dabf904b0\" not found in QueryVolume","TraceId":"6267207f-240e-4d95-8f32-26f7a410bd0d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).ControllerUnpublishVolume\n\t/build/pkg/csi/service/vanilla/controller.go:562\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerUnpublishVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5200\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).controllerUnpublishVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:141\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:88\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerUnpublishVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5202\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-09T10:14:50.911546629Z","caller":"volume/manager.go:433","msg":"failed to delete volume: \"cb25b747-b94b-45ed-8760-1c1dabf904b0\", fault: \"(*types.LocalizedMethodFault)(0xc0006032a0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\\n  Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"0c52d35b\"","TraceId":"720bb3b8-585c-4ae7-bc27-6aa54c512a1c","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/volume.(*defaultManager).DeleteVolume\n\t/build/pkg/common/cns-lib/volume/manager.go:433\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/common.DeleteVolumeUtil\n\t/build/pkg/csi/service/common/vsphereutil.go:349\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:449\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-09T10:14:50.911741459Z","caller":"common/vsphereutil.go:351","msg":"failed to delete disk cb25b747-b94b-45ed-8760-1c1dabf904b0 with error failed to delete volume: \"cb25b747-b94b-45ed-8760-1c1dabf904b0\", fault: \"(*types.LocalizedMethodFault)(0xc0006032a0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\\n  Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"0c52d35b\"","TraceId":"720bb3b8-585c-4ae7-bc27-6aa54c512a1c","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/common.DeleteVolumeUtil\n\t/build/pkg/csi/service/common/vsphereutil.go:351\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:449\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}

Environment:

RaunakShah commented 4 years ago

@gricuk can you list the exact steps you are following before you get into this state?

Do you have Pods running that claim these volumes? If yes, deleting the PV won't detach the volume.

gricuk commented 4 years ago

@gricuk can you list the exact steps you are following before you get into this state?

Do you have Pods running that claim these volumes? If yes, deleting the PV won't detach the volume.

No, I deleted pod and then I try to delete PV. PV deleted successfully but disks not detached from VM.

RaunakShah commented 4 years ago

Is it possible you are running into this issue? https://vsphere-csi-driver.sigs.k8s.io/known_issues.html#issue_5

Anil-YadavK8s commented 4 years ago

@RaunakShah Any workaround how to delete the disk attached to VMs. Because of this issue pod scheduling is getting failed as SCSI controller limitation has been reached.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

gohilankit commented 3 years ago

@gricuk @Anil-YadavK8s - Are you still facing this issue?

gohilankit commented 3 years ago

/assign

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/223#issuecomment-754975120): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.