search

yunify / qingcloud-csi

Kubernetes volume plugin based on CSI specification which support block storage of qingcloud
Apache License 2.0
37 stars 22 forks source link

Cannot find device path of volume #53

Closed wnxn closed 5 years ago

wnxn commented 5 years ago

Logs

  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Warning  FailedScheduling        11m (x25 over 11m)  default-scheduler        pod has unbound immediate PersistentVolumeClaims (repeated 4 times)
  Normal   SuccessfulAttachVolume  11m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-99377a0efc2811e8"
  Warning  FailedMount             33s (x13 over 10m)  kubelet, node2           MountVolume.MountDevice failed for volume "pvc-99377a0efc2811e8" : rpc error: code = Internal desc = Cannot find device path of volume vol-17a9wpqd
  Warning  FailedMount             9s (x5 over 9m10s)  kubelet, node2           Unable to mount volumes for pod "redis-6f5f87dbb5-858rr_kubesphere-system(99cdf74c-fc28-11e8-af86-52546c8e95e0)": timeout expired waiting for volumes to attach or mount for pod "kubesphere-system"/"redis-6f5f87dbb5-858rr". list of unmounted volumes=[redis-pvc]. list of unattached volumes=[redis-pvc default-token-nbvst]
wnxn commented 5 years ago

We should wait longer because the platform a little bit slow.

wnxn commented 5 years ago

Retry in 6 seconds

I1008 06:21:05.680550       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1008 06:21:05.680584       1 rpcserver.go:118] GRPC request: {"node_id":"i-jxbquq8m","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"fsType":"ext4","maxSize":"5000","minSize":"100","replica":"2","stepSize":"50","storage.kubernetes.io/csiProvisionerIdentity":"1570506699179-8081-csi-qingcloud","type":"2"},"volume_id":"vol-wrlo173n"}
I1008 06:21:05.682557       1 controllerserver.go:402] *************** enter ControllerPublishVolume at 2019-10-08 06:21:05 hash dc955fcd ***************
I1008 06:21:05.682570       1 controllerserver.go:427] dc955fcd: Try to lock resource vol-wrlo173n
I1008 06:21:05.682578       1 controllerserver.go:433] dc955fcd: Find volume vol-wrlo173n
I1008 06:21:06.075160       1 controllerserver.go:444] dc955fcd: Find instance i-jxbquq8m
I1008 06:21:06.454859       1 controllerserver.go:471] dc955fcd: Attaching volume vol-wrlo173n to instance i-jxbquq8m in zone ap2a...
I1008 06:21:06.454899       1 qingcloud_manager.go:405] Call IaaS AttachVolume request volume id: vol-wrlo173n, instance id: i-jxbquq8m, zone: ap2a
I1008 06:21:07.629209       1 qingcloud_manager.go:417] Call IaaS WaitJob j-ej8ofwg4hqr
I1008 06:21:07.825047       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I1008 06:21:07.825072       1 rpcserver.go:118] GRPC request: {"node_id":"i-jxbquq8m","volume_id":"vol-k6gaanfw"}
I1008 06:21:07.825668       1 controllerserver.go:514] *************** enter ControllerUnpublishVolume at 2019-10-08 06:21:07 hash 3b3df353 ***************
I1008 06:21:07.825676       1 controllerserver.go:530] Try to lock resource vol-k6gaanfw
I1008 06:21:07.825685       1 controllerserver.go:538] Find volume vol-k6gaanfw
W1008 06:21:08.189908       1 controllerserver.go:546] Volume vol-k6gaanfw is not attached to any instance
I1008 06:21:08.189945       1 controllerserver.go:547] =============== exit ControllerUnpublishVolume at 2019-10-08 06:21:07 hash 3b3df353 ===============
I1008 06:21:08.189954       1 rpcserver.go:123] GRPC response: {}
I1008 06:21:17.974466       1 qingcloud_manager.go:426] Call IaaS AttachVolume vol-wrlo173n on instance i-jxbquq8m succeed
I1008 06:21:18.350778       1 controllerserver.go:492] dc955fcd: Cannot find device path and retry to find volume device vol-wrlo173n
I1008 06:21:19.714765       1 controllerserver.go:492] dc955fcd: Cannot find device path and retry to find volume device vol-wrlo173n
I1008 06:21:22.081262       1 controllerserver.go:492] dc955fcd: Cannot find device path and retry to find volume device vol-wrlo173n
I1008 06:21:25.081707       1 controllerserver.go:498] dc955fcd: Cannot find device path and going to detach volume vol-wrlo173n
I1008 06:21:25.081763       1 qingcloud_manager.go:440] Call IaaS DetachVolume request volume id: vol-wrlo173n, instance id: i-jxbquq8m, zone: ap2a
I1008 06:21:25.543904       1 qingcloud_manager.go:452] Call IaaS WaitJob j-h9pdshbrxt1
I1008 06:21:35.888107       1 qingcloud_manager.go:461] Call IaaS DetachVolume vol-wrlo173n succeed
I1008 06:21:35.888148       1 controllerserver.go:502] =============== exit ControllerPublishVolume at 2019-10-08 06:21:05 hash dc955fcd ===============
E1008 06:21:35.888158       1 rpcserver.go:121] GRPC error: rpc error: code = Internal desc = cannot find device path, volume vol-wrlo173n has been detached, please try attaching to instance i-jxbquq8m again.
I1008 06:21:36.582930       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1008 06:21:36.582963       1 rpcserver.go:118] GRPC request: {"node_id":"i-jxbquq8m","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"fsType":"ext4","maxSize":"5000","minSize":"100","replica":"2","stepSize":"50","storage.kubernetes.io/csiProvisionerIdentity":"1570506699179-8081-csi-qingcloud","type":"2"},"volume_id":"vol-wrlo173n"}
I1008 06:21:36.586239       1 controllerserver.go:402] *************** enter ControllerPublishVolume at 2019-10-08 06:21:36 hash 45607cd0 ***************
I1008 06:21:36.586309       1 controllerserver.go:427] 45607cd0: Try to lock resource vol-wrlo173n
I1008 06:21:36.586322       1 controllerserver.go:433] 45607cd0: Find volume vol-wrlo173n
I1008 06:21:36.957431       1 controllerserver.go:444] 45607cd0: Find instance i-jxbquq8m
I1008 06:21:37.342905       1 controllerserver.go:471] 45607cd0: Attaching volume vol-wrlo173n to instance i-jxbquq8m in zone ap2a...
I1008 06:21:37.342933       1 qingcloud_manager.go:405] Call IaaS AttachVolume request volume id: vol-wrlo173n, instance id: i-jxbquq8m, zone: ap2a
I1008 06:21:38.488648       1 qingcloud_manager.go:417] Call IaaS WaitJob j-mqmshn32twy
I1008 06:21:48.836191       1 controllerserver.go:474] =============== exit ControllerPublishVolume at 2019-10-08 06:21:36 hash 45607cd0 ===============
E1008 06:21:48.836237       1 rpcserver.go:121] GRPC error: rpc error: code = Internal desc = call IaaS WaitJob id Job [j-mqmshn32twy] failed, error:
I1008 06:21:48.980024       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1008 06:21:48.980063       1 rpcserver.go:118] GRPC request: {"node_id":"i-jxbquq8m","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"fsType":"ext4","maxSize":"5000","minSize":"100","replica":"2","stepSize":"50","storage.kubernetes.io/csiProvisionerIdentity":"1570506699179-8081-csi-qingcloud","type":"2"},"volume_id":"vol-wrlo173n"}
I1008 06:21:48.982872       1 controllerserver.go:402] *************** enter ControllerPublishVolume at 2019-10-08 06:21:48 hash a3bebdd7 ***************
I1008 06:21:48.982891       1 controllerserver.go:427] a3bebdd7: Try to lock resource vol-wrlo173n
I1008 06:21:48.982903       1 controllerserver.go:433] a3bebdd7: Find volume vol-wrlo173n
I1008 06:21:49.350527       1 controllerserver.go:444] a3bebdd7: Find instance i-jxbquq8m
I1008 06:21:49.731777       1 controllerserver.go:471] a3bebdd7: Attaching volume vol-wrlo173n to instance i-jxbquq8m in zone ap2a...
I1008 06:21:49.731868       1 qingcloud_manager.go:405] Call IaaS AttachVolume request volume id: vol-wrlo173n, instance id: i-jxbquq8m, zone: ap2a
I1008 06:21:50.088271       1 controllerserver.go:474] =============== exit ControllerPublishVolume at 2019-10-08 06:21:48 hash a3bebdd7 ===============
E1008 06:21:50.088302       1 rpcserver.go:121] GRPC error: rpc error: code = Internal desc = QingCloud Error: Code (1400), Message (PermissionDenied, you can only attach [10] volumes to one instance at most)
I1008 06:21:58.881534       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1008 06:21:58.881565       1 rpcserver.go:118] GRPC request: {"node_id":"i-jxbquq8m","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"fsType":"ext4","maxSize":"5000","minSize":"100","replica":"2","stepSize":"50","storage.kubernetes.io/csiProvisionerIdentity":"1570506699179-8081-csi-qingcloud","type":"2"},"volume_id":"vol-wrlo173n"}
I1008 06:21:58.885443       1 controllerserver.go:402] *************** enter ControllerPublishVolume at 2019-10-08 06:21:58 hash 9bb9242c ***************
I1008 06:21:58.885466       1 controllerserver.go:427] 9bb9242c: Try to lock resource vol-wrlo173n
I1008 06:21:58.885490       1 controllerserver.go:433] 9bb9242c: Find volume vol-wrlo173n
I1008 06:21:59.263235       1 controllerserver.go:444] 9bb9242c: Find instance i-jxbquq8m
I1008 06:21:59.647333       1 controllerserver.go:471] 9bb9242c: Attaching volume vol-wrlo173n to instance i-jxbquq8m in zone ap2a...
I1008 06:21:59.647371       1 qingcloud_manager.go:405] Call IaaS AttachVolume request volume id: vol-wrlo173n, instance id: i-jxbquq8m, zone: ap2a
I1008 06:21:59.999087       1 controllerserver.go:474] =============== exit ControllerPublishVolume at 2019-10-08 06:21:58 hash 9bb9242c ===============
E1008 06:21:59.999141       1 rpcserver.go:121] GRPC error: rpc error: code = Internal desc = QingCloud Error: Code (1400), Message (PermissionDenied, you can only attach [10] volumes to one instance at most)
I1008 06:22:40.008164       1 rpcserver.go:117] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I1008 06:22:40.008195       1 rpcserver.go:118] GRPC request: {"node_id":"i-jxbquq8m","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"fsType":"ext4","maxSize":"5000","minSize":"100","replica":"2","stepSize":"50","storage.kubernetes.io/csiProvisionerIdentity":"1570506699179-8081-csi-qingcloud","type":"2"},"volume_id":"vol-wrlo173n"}
I1008 06:22:40.009913       1 controllerserver.go:402] *************** enter ControllerPublishVolume at 2019-10-08 06:22:40 hash 5d5775ba ***************
I1008 06:22:40.009925       1 controllerserver.go:427] 5d5775ba: Try to lock resource vol-wrlo173n
I1008 06:22:40.009936       1 controllerserver.go:433] 5d5775ba: Find volume vol-wrlo173n
I1008 06:22:40.371562       1 controllerserver.go:444] 5d5775ba: Find instance i-jxbquq8m
I1008 06:22:40.746050       1 controllerserver.go:471] 5d5775ba: Attaching volume vol-wrlo173n to instance i-jxbquq8m in zone ap2a...
I1008 06:22:40.746087       1 qingcloud_manager.go:405] Call IaaS AttachVolume request volume id: vol-wrlo173n, instance id: i-jxbquq8m, zone: ap2a
I1008 06:22:41.101132       1 controllerserver.go:474] =============== exit ControllerPublishVolume at 2019-10-08 06:22:40 hash 5d5775ba ===============
E1008 06:22:41.101171       1 rpcserver.go:121] GRPC error: rpc error: code = Internal desc = QingCloud Error: Code (1400), Message (PermissionDenied, you can only attach [10] volumes to one instance at most)