search

ctrox / csi-s3

A Container Storage Interface for S3
Apache License 2.0
758 stars 167 forks source link

Pod created failed: rpc error: code = Unknown desc = The specified key does not exist. #56

Open wenhuwang opened 2 years ago

wenhuwang commented 2 years ago

csi components description:

$ kubectl  -n kube-system get pods
csi-attacher-s3-0                          1/1     Running   2          32m
csi-provisioner-s3-0                       2/2     Running   0          32m
csi-s3-mf6pj                               2/2     Running   0          10m
csi-s3-mnczm                               2/2     Running   0          11m
csi-s3-vfmw5                               2/2     Running   0          11m

$ kubectl -n kube-system logs -f csi-s3-mnczm csi-s3 
I0726 03:40:46.556126       1 s3-driver.go:80] Driver: ch.ctrox.csi.s3-driver 
I0726 03:40:46.556378       1 s3-driver.go:81] Version: v1.1.1 
I0726 03:40:46.556392       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0726 03:40:46.556401       1 driver.go:93] Enabling volume access mode: SINGLE_NODE_WRITER
I0726 03:40:46.557503       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0726 03:40:46.591459       1 utils.go:97] GRPC call: /csi.v1.Identity/Probe
I0726 03:40:46.591495       1 utils.go:98] GRPC request: {}
I0726 03:40:46.592577       1 utils.go:103] GRPC response: {}
I0726 03:40:46.594101       1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0726 03:40:46.594113       1 utils.go:98] GRPC request: {}
I0726 03:40:46.594610       1 identityserver-default.go:32] Using default GetPluginInfo
I0726 03:40:46.594616       1 utils.go:103] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"}
I0726 03:40:46.595564       1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0726 03:40:46.595579       1 utils.go:98] GRPC request: {}
I0726 03:40:46.596134       1 identityserver-default.go:53] Using default capabilities
I0726 03:40:46.596141       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}}]}
I0726 03:40:46.597643       1 utils.go:97] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0726 03:40:46.597657       1 utils.go:98] GRPC request: {}
I0726 03:40:46.598341       1 controllerserver-default.go:62] Using default ControllerGetCapabilities
I0726 03:40:46.598348       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0726 03:40:46.599948       1 utils.go:97] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0726 03:40:46.599958       1 utils.go:98] GRPC request: {}
I0726 03:40:46.601157       1 controllerserver-default.go:62] Using default ControllerGetCapabilities
I0726 03:40:46.601164       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0726 03:40:47.121684       1 utils.go:97] GRPC call: /csi.v1.Identity/GetPluginInfo
I0726 03:40:47.121704       1 utils.go:98] GRPC request: {}
I0726 03:40:47.122214       1 identityserver-default.go:32] Using default GetPluginInfo
I0726 03:40:47.122220       1 utils.go:103] GRPC response: {"name":"ch.ctrox.csi.s3-driver","vendor_version":"v1.1.1"}
I0726 03:40:48.441612       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetInfo
I0726 03:40:48.441648       1 utils.go:98] GRPC request: {}
I0726 03:40:48.442169       1 nodeserver-default.go:40] Using default NodeGetInfo
I0726 03:40:48.442176       1 utils.go:103] GRPC response: {"node_id":"shtl009063226"}
I0726 03:41:18.921095       1 utils.go:97] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0726 03:41:18.921114       1 utils.go:98] GRPC request: {}
I0726 03:41:18.921505       1 utils.go:103] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}}]}
I0726 03:41:18.928505       1 utils.go:97] GRPC call: /csi.v1.Node/NodeStageVolume
I0726 03:41:18.928521       1 utils.go:98] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bb66f0cc-e6c9-4bc9-9662-a7bd40b2dc09/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"mounter":"rclone","storage.kubernetes.io/csiProvisionerIdentity":"1627269615740-8081-ch.ctrox.csi.s3-driver"},"volume_id":"pvc-bb66f0cc-e6c9-4bc9-9662-a7bd40b2dc09"}
E0726 03:41:18.953948       1 utils.go:101] GRPC error: The specified key does not exist.

examples description:

$ kubectl  get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
csi-s3-pvc           Bound    pvc-bb66f0cc-e6c9-4bc9-9662-a7bd40b2dc09   5Gi        RWO            csi-s3                 14m

$ kubectl  get pod
NAME                                                             READY   STATUS              RESTARTS   AGE
csi-s3-test-nginx                                                0/1     ContainerCreating   0          14m

$ kubectl  describe pods csi-s3-test-nginx
...
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               17m                  default-scheduler        Successfully assigned default/csi-s3-test-nginx to shtl009063226
  Normal   SuccessfulAttachVolume  17m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-bb66f0cc-e6c9-4bc9-9662-a7bd40b2dc09"
  Warning  FailedMount             4m26s (x6 over 15m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[webroot default-token-pxlf7]: timed out waiting for the condition
  Warning  FailedMount             2m10s                kubelet                  Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[default-token-pxlf7 webroot]: timed out waiting for the condition
  Warning  FailedMount             82s (x16 over 17m)   kubelet                  MountVolume.MountDevice failed for volume "pvc-bb66f0cc-e6c9-4bc9-9662-a7bd40b2dc09" : rpc error: code = Unknown desc = The specified key does not exist.

Thanks for any help.

georgethegreatat commented 2 years ago

Hi guys. Faced same issue with s3fs :

Warning | FailedMount | Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[webroot default-token-s4cs9]: timed out waiting for the condition | a minute ago
Warning | FailedMount | MountVolume.SetUp failed for volume "pvc-287b3a57-4c6c-46e4-81e4-4dc651ee7a70" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc = stat /var/lib/kubelet/pods/8f17a3e0-5a04-472e-ad65-c9f61a3c8f5e/volumes/kubernetes.io~csi/pvc-287b3a57-4c6c-46e4-81e4-4dc651ee7a70/mount: software caused connection abort | 2 minutes ago
Warning | FailedMount | Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[default-token-s4cs9 webroot]: timed out waiting for the condition | 8 minutes ago
Normal | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-287b3a57-4c6c-46e4-81e4-4dc651ee7a70"

RCLONE dont working too. Faced issue when i can't write or read the files from s3.

Can anybody help me with this ?

wenhuwang commented 2 years ago

Hi guys. Faced same issue with s3fs :


Warning | FailedMount | Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[webroot default-token-s4cs9]: timed out waiting for the condition | a minute ago

Warning | FailedMount | MountVolume.SetUp failed for volume "pvc-287b3a57-4c6c-46e4-81e4-4dc651ee7a70" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc = stat /var/lib/kubelet/pods/8f17a3e0-5a04-472e-ad65-c9f61a3c8f5e/volumes/kubernetes.io~csi/pvc-287b3a57-4c6c-46e4-81e4-4dc651ee7a70/mount: software caused connection abort | 2 minutes ago

Warning | FailedMount | Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[default-token-s4cs9 webroot]: timed out waiting for the condition | 8 minutes ago

Normal | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-287b3a57-4c6c-46e4-81e4-4dc651ee7a70"

RCLONE dont working too. Faced issue when i can't write or read the files from s3.

Can anybody help me with this ?

I change reclone to s3fs mode,this issues has been solved.

georgethegreatat commented 2 years ago

@wenhuwang as i told before I faced this issue with s3fs ;/

wenhuwang commented 2 years ago

@wenhuwang as i told before I faced this issue with s3fs ;/

Check you yaml with the repo directory deploy/kubernetes,and use s3fs.

Bowen0729 commented 2 years ago

@wenhuwang as i told before I faced this issue with s3fs ;/

Have you solved it?

CallMeLaNN commented 1 year ago

I'm using latest managed Kubernetes v1.25.4 Digital Ocean using their Spaces for the s3, with mounter rclone. I got the same error as OP. After a day trying an error, I managed to make it work and reproduce the error:

GRPC call: /csi.v1.Node/NodeStageVolume
GRPC error: The specified key does not exist.

The problem is using in StorageClass with usePrefix: "true" and prefix contain / path separator. Tested using rclone and it is working without having / in prefix.

Working: prefix: "parent-child" Not working: prefix: "parent/child"

If contain /, it create the parent and child subfolder correctly (can provision and attach) but at runtime it can't find the path.

# 1. usePrefix with path separator
metadata: parent/child/.metadata.json
mounted s3 path: (don't know, only the metadata file created)
# 2. usePrefix without path separator
metadata: parent-child/.metadata.json
mounted s3 path: parent-child/*
# 3. without usePrefix
metadata: pvc-xxx/.metadata.json
mounted s3 path: pvc-xxx/csi-fs/*

@ctrox It is worth to note that if I usePrefix without path separator (no 2 above), from the mounted folder inside pod, I can see .metadata.json!. The folder structure is not consistent. I already run this code but since go is no my everyday use, I can't debug it easily. I hope this info will help to fix this.