search

yunify / qingcloud-csi

Kubernetes volume plugin based on CSI specification which support block storage of qingcloud
Apache License 2.0
37 stars 22 forks source link

Keep resize idempotent during several same calls #152

Closed wnxn closed 4 years ago

wnxn commented 4 years ago

What happened: Controller Resize Finished but external resizer calls failed.

What you expected to happen: Controller Resize Finished and external resizer calls succeed.

How to reproduce it (as minimally and precisely as possible):

  1. create 100G high capacity pvc
  2. mount on pod
  3. expand pvc to 200G

Anything else we need to know?:

I1101 13:26:25.199755       1 qingcloud_manager.go:461] Call IaaS DetachVolume vol-0svc0trn succeed
I1101 13:26:25.199802       1 controllerserver.go:574] =============== exit ControllerUnpublishVolume at 2019-11-01 13:26:13 hash 7ff9cbde ===============
I1101 13:26:25.199815       1 rpcserver.go:123] GRPC response: {}
I1101 13:26:28.015898       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1101 13:26:28.015920       1 rpcserver.go:118] GRPC request: {"capacity_range":{"required_bytes":214748364800},"volume_id":"vol-0svc0trn"}
I1101 13:26:28.016957       1 controllerserver.go:632] *************** enter ControllerExpandVolume at 2019-11-01 13:26:28 hash 6f662f30 ***************
I1101 13:26:28.016963       1 controllerserver.go:643] Try to lock resource vol-0svc0trn
I1101 13:26:28.016970       1 controllerserver.go:649] Find volume vol-0svc0trn
I1101 13:26:28.398298       1 controllerserver.go:670] 6f662f30: Succeed to get volume [vol-0svc0trn] type [HighCapacity]
I1101 13:26:28.398344       1 controllerserver.go:684] Try to expand volume vol-0svc0trn
I1101 13:26:28.398355       1 qingcloud_manager.go:469] Call IaaS ResizeVolume request volume vol-0svc0trn size 200 Gib in zone [ap2a]
I1101 13:26:29.684386       1 qingcloud_manager.go:485] Call IaaS WaitJob j-2k2f3m9o3ut
I1101 13:26:44.916781       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1101 13:26:44.916807       1 rpcserver.go:118] GRPC request: {"capacity_range":{"required_bytes":214748364800},"volume_id":"vol-0svc0trn"}
I1101 13:26:44.917862       1 controllerserver.go:632] *************** enter ControllerExpandVolume at 2019-11-01 13:26:44 hash 82f29790 ***************
I1101 13:26:44.917870       1 controllerserver.go:643] Try to lock resource vol-0svc0trn
I1101 13:26:44.917883       1 controllerserver.go:645] =============== exit ControllerExpandVolume at 2019-11-01 13:26:44 hash 82f29790 ===============
E1101 13:26:44.917889       1 rpcserver.go:121] GRPC error: rpc error: code = Aborted desc = already an operation for the specified resource vol-0svc0trn
I1101 13:26:50.028534       1 qingcloud_manager.go:494] Call IaaS ResizeVolume id vol-0svc0trn size 200 succeed
I1101 13:26:50.028571       1 controllerserver.go:689] Succeed to resize volume vol-0svc0trn
I1101 13:26:50.028614       1 controllerserver.go:696] =============== exit ControllerExpandVolume at 2019-11-01 13:26:28 hash 6f662f30 ===============
I1101 13:26:50.028626       1 rpcserver.go:123] GRPC response: {"capacity_bytes":214748364800,"node_expansion_required":true}
I1101 13:26:52.017476       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1101 13:26:52.017502       1 rpcserver.go:118] GRPC request: {"capacity_range":{"required_bytes":214748364800},"volume_id":"vol-0svc0trn"}
I1101 13:26:52.018700       1 controllerserver.go:632] *************** enter ControllerExpandVolume at 2019-11-01 13:26:52 hash fc060065 ***************
I1101 13:26:52.018706       1 controllerserver.go:643] Try to lock resource vol-0svc0trn
I1101 13:26:52.018713       1 controllerserver.go:649] Find volume vol-0svc0trn
I1101 13:26:52.382513       1 controllerserver.go:670] fc060065: Succeed to get volume [vol-0svc0trn] type [HighCapacity]
I1101 13:26:52.382551       1 controllerserver.go:684] Try to expand volume vol-0svc0trn
I1101 13:26:52.382560       1 qingcloud_manager.go:469] Call IaaS ResizeVolume request volume vol-0svc0trn size 200 Gib in zone [ap2a]
E1101 13:26:52.742026       1 controllerserver.go:686] Failed to resize volume vol-0svc0trn, error: QingCloud Error: Code (1400), Message (PermissionDenied, volume [vol-0svc0trn]'s new size [200 GB] should larger than the old size [200 GB])
I1101 13:26:52.742081       1 controllerserver.go:694] =============== exit ControllerExpandVolume at 2019-11-01 13:26:52 hash fc060065 ===============
E1101 13:26:52.742090       1 rpcserver.go:121] GRPC error: rpc error: code = Internal desc = QingCloud Error: Code (1400), Message (PermissionDenied, volume [vol-0svc0trn]'s new size [200 GB] should larger than the old size [200 GB])
I1101 13:27:05.315853       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1101 13:27:05.315879       1 rpcserver.go:118] GRPC request: {"capacity_range":{"required_bytes":214748364800},"volume_id":"vol-0svc0trn"}
I1101 13:27:05.318213       1 controllerserver.go:632] *************** enter ControllerExpandVolume at 2019-11-01 13:27:05 hash cb326096 ***************
I1101 13:27:05.318223       1 controllerserver.go:643] Try to lock resource vol-0svc0trn
I1101 13:27:05.318231       1 controllerserver.go:649] Find volume vol-0svc0trn
I1101 13:27:05.683735       1 controllerserver.go:670] cb326096: Succeed to get volume [vol-0svc0trn] type [HighCapacity]
I1101 13:27:05.683764       1 controllerserver.go:684] Try to expand volume vol-0svc0trn
I1101 13:27:05.683772       1 qingcloud_manager.go:469] Call IaaS ResizeVolume request volume vol-0svc0trn size 200 Gib in zone [ap2a]
E1101 13:27:06.049565       1 controllerserver.go:686] Failed to resize volume vol-0svc0trn, error: QingCloud Error: Code (1400), Message (PermissionDenied, volume [vol-0svc0trn]'s new size [200 GB] should larger than the old size [200 GB])

Environment: