search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.28k stars 547 forks source link

Pod creation fails with "staging path does not exists on node" #1067

Closed chris-sanders closed 4 years ago

chris-sanders commented 4 years ago

Describe the bug

Trying to setup ceph-csi with microk8s against a ceph cluster, just rbd for now. The plugin installs, a pvc/pv is created and be seen in ceph, but the pod stalls with the missing staging path error in the title. The path is indeed missing when I look on the worker, but I can't figure out where/how it was suppose to be created.

Environment details

Steps to reproduce

Steps to reproduce the behavior:

  1. Setup details: sudo snap install microk8s --channel=1.18/stable
  2. vim /var/snap/microk8s/current/args/kube-apiserver Add --allow-privileged
  3. Modify file paths in csi-rbdplugin.yaml in deploy folder with snap paths: https://pastebin.ubuntu.com/p/YRPT7sJgHV/ Modified lines have the original path commented out next to it
  4. Modify the csi-configmap with ceph credentials
  5. Run plugin-deploy.sh in examples folder
  6. apply pvc.yaml
  7. apply pod.yaml
  8. Pod throws error shown below

Actual results

Pod never completes

Expected behavior

Pod is created with the rbd volume

Logs

$ kubectl logs pod/csi-rbdplugin-provisioner-86b4f6d8c5-lb7c6 csi-rbdplugin I0515 01:19:24.530027 1 cephcsi.go:117] Driver version: v2.1.1 and Git version: 9022d899eb6fd464f0be33701d8160ecd1317467 I0515 01:19:24.530280 1 cephcsi.go:144] Initial PID limit is set to -1 I0515 01:19:24.530366 1 cephcsi.go:153] Reconfigured PID limit to -1 (max) I0515 01:19:24.530377 1 cephcsi.go:172] Starting driver type: rbd with name: rbd.csi.ceph.com I0515 01:19:24.530625 1 driver.go:88] Enabling controller service capability: CREATE_DELETE_VOLUME I0515 01:19:24.530636 1 driver.go:88] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0515 01:19:24.530645 1 driver.go:88] Enabling controller service capability: CLONE_VOLUME I0515 01:19:24.530651 1 driver.go:88] Enabling controller service capability: EXPAND_VOLUME I0515 01:19:24.530656 1 driver.go:99] Enabling volume access mode: SINGLE_NODE_WRITER I0515 01:19:24.530662 1 driver.go:99] Enabling volume access mode: MULTI_NODE_MULTI_WRITER I0515 01:19:24.530979 1 server.go:116] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi-provisioner.sock", Net:"unix"} I0515 01:19:24.776836 1 utils.go:159] ID: 1 GRPC call: /csi.v1.Identity/Probe I0515 01:19:24.776862 1 utils.go:160] ID: 1 GRPC request: {} I0515 01:19:24.778214 1 utils.go:165] ID: 1 GRPC response: {} I0515 01:19:24.783336 1 utils.go:159] ID: 2 GRPC call: /csi.v1.Identity/GetPluginInfo I0515 01:19:24.783352 1 utils.go:160] ID: 2 GRPC request: {} I0515 01:19:24.785854 1 identityserver-default.go:37] ID: 2 Using default GetPluginInfo I0515 01:19:24.785870 1 utils.go:165] ID: 2 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"} I0515 01:19:24.788756 1 utils.go:159] ID: 3 GRPC call: /csi.v1.Identity/GetPluginCapabilities I0515 01:19:24.788773 1 utils.go:160] ID: 3 GRPC request: {} I0515 01:19:24.789490 1 utils.go:165] ID: 3 GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]} I0515 01:19:24.795930 1 utils.go:159] ID: 4 GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0515 01:19:24.795955 1 utils.go:160] ID: 4 GRPC request: {} I0515 01:19:24.796543 1 controllerserver-default.go:63] ID: 4 Using default ControllerGetCapabilities I0515 01:19:24.796551 1 utils.go:165] ID: 4 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]} I0515 01:19:24.803803 1 utils.go:159] ID: 5 GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0515 01:19:24.803817 1 utils.go:160] ID: 5 GRPC request: {} I0515 01:19:24.805595 1 controllerserver-default.go:63] ID: 5 Using default ControllerGetCapabilities I0515 01:19:24.805612 1 utils.go:165] ID: 5 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]} I0515 01:19:24.860814 1 utils.go:159] ID: 6 GRPC call: /csi.v1.Identity/Probe I0515 01:19:24.860846 1 utils.go:160] ID: 6 GRPC request: {} I0515 01:19:24.861640 1 utils.go:165] ID: 6 GRPC response: {} I0515 01:19:24.864142 1 utils.go:159] ID: 7 GRPC call: /csi.v1.Identity/GetPluginInfo I0515 01:19:24.864159 1 utils.go:160] ID: 7 GRPC request: {} I0515 01:19:24.864883 1 identityserver-default.go:37] ID: 7 Using default GetPluginInfo I0515 01:19:24.864898 1 utils.go:165] ID: 7 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"} I0515 01:19:24.869363 1 utils.go:159] ID: 8 GRPC call: /csi.v1.Identity/GetPluginCapabilities I0515 01:19:24.869389 1 utils.go:160] ID: 8 GRPC request: {} I0515 01:19:24.871253 1 utils.go:165] ID: 8 GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]} I0515 01:19:24.876734 1 utils.go:159] ID: 9 GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0515 01:19:24.876752 1 utils.go:160] ID: 9 GRPC request: {} I0515 01:19:24.877319 1 controllerserver-default.go:63] ID: 9 Using default ControllerGetCapabilities I0515 01:19:24.877328 1 utils.go:165] ID: 9 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]} I0515 01:19:25.128644 1 utils.go:159] ID: 10 GRPC call: /csi.v1.Identity/Probe I0515 01:19:25.128663 1 utils.go:160] ID: 10 GRPC request: {} I0515 01:19:25.129208 1 utils.go:165] ID: 10 GRPC response: {} I0515 01:19:25.133847 1 utils.go:159] ID: 11 GRPC call: /csi.v1.Identity/GetPluginInfo I0515 01:19:25.133875 1 utils.go:160] ID: 11 GRPC request: {} I0515 01:19:25.134858 1 identityserver-default.go:37] ID: 11 Using default GetPluginInfo I0515 01:19:25.134876 1 utils.go:165] ID: 11 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"} I0515 01:19:25.137355 1 utils.go:159] ID: 12 GRPC call: /csi.v1.Identity/GetPluginCapabilities I0515 01:19:25.137375 1 utils.go:160] ID: 12 GRPC request: {} I0515 01:19:25.138035 1 utils.go:165] ID: 12 GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]} I0515 01:19:25.146660 1 utils.go:159] ID: 13 GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0515 01:19:25.146696 1 utils.go:160] ID: 13 GRPC request: {} I0515 01:19:25.150550 1 controllerserver-default.go:63] ID: 13 Using default ControllerGetCapabilities I0515 01:19:25.150581 1 utils.go:165] ID: 13 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]} I0515 01:19:25.272278 1 utils.go:159] ID: 14 GRPC call: /csi.v1.Identity/GetPluginInfo I0515 01:19:25.272313 1 utils.go:160] ID: 14 GRPC request: {} I0515 01:19:25.273149 1 identityserver-default.go:37] ID: 14 Using default GetPluginInfo I0515 01:19:25.273161 1 utils.go:165] ID: 14 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"} I0515 01:19:25.275900 1 utils.go:159] ID: 15 GRPC call: /csi.v1.Identity/Probe I0515 01:19:25.275924 1 utils.go:160] ID: 15 GRPC request: {} I0515 01:19:25.276741 1 utils.go:165] ID: 15 GRPC response: {} I0515 01:19:25.279375 1 utils.go:159] ID: 16 GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0515 01:19:25.279393 1 utils.go:160] ID: 16 GRPC request: {} I0515 01:19:25.280111 1 controllerserver-default.go:63] ID: 16 Using default ControllerGetCapabilities I0515 01:19:25.280123 1 utils.go:165] ID: 16 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]} I0515 01:20:24.919719 1 utils.go:159] ID: 17 GRPC call: /csi.v1.Identity/Probe I0515 01:20:24.919744 1 utils.go:160] ID: 17 GRPC request: {} I0515 01:20:24.920660 1 utils.go:165] ID: 17 GRPC response: {} I0515 01:21:24.919765 1 utils.go:159] ID: 18 GRPC call: /csi.v1.Identity/Probe I0515 01:21:24.919794 1 utils.go:160] ID: 18 GRPC request: {} I0515 01:21:24.920746 1 utils.go:165] ID: 18 GRPC response: {} I0515 01:22:24.919593 1 utils.go:159] ID: 19 GRPC call: /csi.v1.Identity/Probe I0515 01:22:24.919616 1 utils.go:160] ID: 19 GRPC request: {} I0515 01:22:24.920234 1 utils.go:165] ID: 19 GRPC response: {} I0515 01:23:24.919639 1 utils.go:159] ID: 20 GRPC call: /csi.v1.Identity/Probe I0515 01:23:24.919669 1 utils.go:160] ID: 20 GRPC request: {} I0515 01:23:24.920493 1 utils.go:165] ID: 20 GRPC response: {} I0515 01:24:24.919587 1 utils.go:159] ID: 21 GRPC call: /csi.v1.Identity/Probe I0515 01:24:24.919614 1 utils.go:160] ID: 21 GRPC request: {} I0515 01:24:24.920467 1 utils.go:165] ID: 21 GRPC response: {} I0515 01:25:24.919544 1 utils.go:159] ID: 22 GRPC call: /csi.v1.Identity/Probe I0515 01:25:24.919565 1 utils.go:160] ID: 22 GRPC request: {} I0515 01:25:24.920243 1 utils.go:165] ID: 22 GRPC response: {} I0515 01:26:24.919605 1 utils.go:159] ID: 23 GRPC call: /csi.v1.Identity/Probe I0515 01:26:24.919631 1 utils.go:160] ID: 23 GRPC request: {} I0515 01:26:24.920467 1 utils.go:165] ID: 23 GRPC response: {} I0515 01:27:24.919649 1 utils.go:159] ID: 24 GRPC call: /csi.v1.Identity/Probe I0515 01:27:24.919676 1 utils.go:160] ID: 24 GRPC request: {} I0515 01:27:24.920519 1 utils.go:165] ID: 24 GRPC response: {} I0515 01:28:24.919597 1 utils.go:159] ID: 25 GRPC call: /csi.v1.Identity/Probe I0515 01:28:24.919623 1 utils.go:160] ID: 25 GRPC request: {} I0515 01:28:24.920392 1 utils.go:165] ID: 25 GRPC response: {} I0515 01:29:24.919569 1 utils.go:159] ID: 26 GRPC call: /csi.v1.Identity/Probe I0515 01:29:24.919594 1 utils.go:160] ID: 26 GRPC request: {} I0515 01:29:24.920399 1 utils.go:165] ID: 26 GRPC response: {} I0515 01:30:24.919469 1 utils.go:159] ID: 27 GRPC call: /csi.v1.Identity/Probe I0515 01:30:24.919508 1 utils.go:160] ID: 27 GRPC request: {} I0515 01:30:24.920487 1 utils.go:165] ID: 27 GRPC response: {} I0515 01:31:24.919621 1 utils.go:159] ID: 28 GRPC call: /csi.v1.Identity/Probe I0515 01:31:24.919648 1 utils.go:160] ID: 28 GRPC request: {} I0515 01:31:24.920525 1 utils.go:165] ID: 28 GRPC response: {} I0515 01:32:24.919728 1 utils.go:159] ID: 29 GRPC call: /csi.v1.Identity/Probe I0515 01:32:24.919754 1 utils.go:160] ID: 29 GRPC request: {} I0515 01:32:24.920655 1 utils.go:165] ID: 29 GRPC response: {} I0515 01:33:24.919512 1 utils.go:159] ID: 30 GRPC call: /csi.v1.Identity/Probe I0515 01:33:24.919536 1 utils.go:160] ID: 30 GRPC request: {} I0515 01:33:24.920273 1 utils.go:165] ID: 30 GRPC response: {}

$ kubectl logs pod/csi-rbdplugin-provisioner-86b4f6d8c5-lb7c6 csi-provisioner I0515 01:19:22.079083 1 feature_gate.go:216] feature gates: &{map[Topology:true]} I0515 01:19:22.079145 1 csi-provisioner.go:98] Version: v1.4.0-0-g1d9bad3 I0515 01:19:22.079164 1 csi-provisioner.go:112] Building kube configs for running in cluster... I0515 01:19:22.092898 1 connection.go:151] Connecting to unix:///csi/csi-provisioner.sock I0515 01:19:24.859241 1 connection.go:261] Probing CSI driver for readiness I0515 01:19:24.859270 1 connection.go:180] GRPC call: /csi.v1.Identity/Probe I0515 01:19:24.859284 1 connection.go:181] GRPC request: {} I0515 01:19:24.862442 1 connection.go:183] GRPC response: {} I0515 01:19:24.863159 1 connection.go:184] GRPC error: I0515 01:19:24.863173 1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo I0515 01:19:24.863181 1 connection.go:181] GRPC request: {} I0515 01:19:24.865738 1 connection.go:183] GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"} I0515 01:19:24.866540 1 connection.go:184] GRPC error: I0515 01:19:24.866558 1 csi-provisioner.go:152] Detected CSI driver rbd.csi.ceph.com I0515 01:19:24.866568 1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginCapabilities I0515 01:19:24.866576 1 connection.go:181] GRPC request: {} I0515 01:19:24.873567 1 connection.go:183] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]} I0515 01:19:24.875938 1 connection.go:184] GRPC error: I0515 01:19:24.875954 1 connection.go:180] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0515 01:19:24.875961 1 connection.go:181] GRPC request: {} I0515 01:19:24.880690 1 connection.go:183] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]} I0515 01:19:24.884481 1 connection.go:184] GRPC error: I0515 01:19:24.885049 1 controller.go:680] Using saving PVs to API server in background I0515 01:19:24.885223 1 leaderelection.go:241] attempting to acquire leader lease default/rbd-csi-ceph-com... I0515 01:19:24.890200 1 leaderelection.go:350] lock is held by csi-rbdplugin-provisioner-86b4f6d8c5-5tv6j and has not yet expired I0515 01:19:24.890239 1 leaderelection.go:246] failed to acquire lease default/rbd-csi-ceph-com I0515 01:19:24.890255 1 leader_election.go:172] new leader detected, current leader: csi-rbdplugin-provisioner-86b4f6d8c5-5tv6j I0515 01:19:35.536029 1 leaderelection.go:350] lock is held by csi-rbdplugin-provisioner-86b4f6d8c5-5tv6j and has not yet expired I0515 01:19:35.536053 1 leaderelection.go:246] failed to acquire lease default/rbd-csi-ceph-com I0515 01:19:44.528850 1 leaderelection.go:251] successfully acquired lease default/rbd-csi-ceph-com I0515 01:19:44.528949 1 leader_election.go:172] new leader detected, current leader: csi-rbdplugin-provisioner-86b4f6d8c5-lb7c6 I0515 01:19:44.528973 1 leader_election.go:165] became leader, starting I0515 01:19:44.529013 1 controller.go:770] Starting provisioner controller rbd.csi.ceph.com_csi-rbdplugin-provisioner-86b4f6d8c5-lb7c6_acb9d837-4912-4068-ae6d-7d9fe39a5896! I0515 01:19:44.529120 1 volume_store.go:97] Starting save volume queue I0515 01:19:44.529419 1 reflector.go:122] Starting reflector v1.PersistentVolumeClaim (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:801 I0515 01:19:44.529443 1 reflector.go:122] Starting reflector v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:804 I0515 01:19:44.529456 1 reflector.go:160] Listing and watching v1.PersistentVolumeClaim from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:801 I0515 01:19:44.529458 1 reflector.go:122] Starting reflector v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:807 I0515 01:19:44.529467 1 reflector.go:160] Listing and watching v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:804 I0515 01:19:44.529482 1 reflector.go:160] Listing and watching v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:807 I0515 01:19:44.533832 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:19:44.629279 1 shared_informer.go:177] caches populated I0515 01:19:44.629658 1 controller.go:819] Started provisioner controller rbd.csi.ceph.com_csi-rbdplugin-provisioner-86b4f6d8c5-lb7c6_acb9d837-4912-4068-ae6d-7d9fe39a5896! I0515 01:19:49.539512 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:19:54.550073 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:19:59.555314 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:04.560593 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:09.565959 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:14.571475 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:19.576857 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:24.582119 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:29.587070 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:34.592518 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:39.597881 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:44.603562 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:49.608998 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:54.614427 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:20:59.619767 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:04.625043 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:09.630837 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:14.636163 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:19.641407 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:24.646704 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:29.652067 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:34.657369 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:39.663123 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:44.668401 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:49.673548 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:54.678908 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:21:59.684018 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:04.689263 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:09.694933 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:14.700029 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:19.705222 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:24.710370 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:29.717686 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:34.723431 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:39.729948 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:44.735962 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:49.741581 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:54.747374 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:22:59.752744 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:04.758323 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:09.764153 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:14.769397 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:19.775023 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:24.780441 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:29.785720 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:34.791487 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:39.797267 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:44.801950 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:49.808157 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:54.813659 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:23:59.819255 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:04.825021 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:09.830440 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:14.836056 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:19.841534 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:24.847008 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:29.852151 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:34.857652 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:39.863273 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:44.868760 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:49.874381 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:54.879965 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:24:59.885338 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:03.539365 1 reflector.go:389] sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:804: Watch close - v1.PersistentVolume total 0 items received I0515 01:25:04.890712 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:09.896092 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:14.901319 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:19.906884 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:24.912839 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:29.918289 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:34.923586 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:39.929091 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:44.934632 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:49.940065 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:54.944752 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:25:59.950550 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:04.956099 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:09.961849 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:14.967393 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:19.973120 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:24.978868 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:29.984061 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:34.989119 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:39.994078 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:44.999779 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:50.005460 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:26:51.533635 1 reflector.go:389] sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:807: Watch close - v1.StorageClass total 0 items received I0515 01:26:55.010573 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:00.015553 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:05.021174 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:10.026624 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:15.031991 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:20.037378 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:25.043123 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:30.048504 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:35.053964 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:40.059628 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:45.065276 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:50.070905 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:27:55.076220 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:00.081295 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:05.088790 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:10.093837 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:10.535268 1 reflector.go:389] sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:801: Watch close - v1.PersistentVolumeClaim total 0 items received I0515 01:28:15.099482 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:20.104558 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:25.109958 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:30.115380 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:35.120689 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:40.126447 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:45.131664 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:50.136731 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:28:55.141912 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:00.147560 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:05.152749 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:10.158209 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:15.163478 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:20.168501 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:25.173697 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:30.178740 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:35.183780 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:40.188605 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:45.193807 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:50.199183 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:29:55.204157 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:00.209310 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:05.214170 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:10.219402 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:15.224301 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:20.229159 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:25.234490 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:30.239578 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:35.244622 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:40.250178 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:45.255485 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:49.540781 1 reflector.go:389] sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:804: Watch close - v1.PersistentVolume total 0 items received I0515 01:30:50.261199 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:30:55.266490 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:00.271942 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:05.277428 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:10.283031 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:15.288249 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:20.293406 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:25.298136 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:30.303363 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:35.308560 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:40.313696 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:45.319604 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:50.324833 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:31:55.330316 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:32:00.335535 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:32:05.340795 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com I0515 01:32:10.345617 1 leaderelection.go:282] successfully renewed lease default/rbd-csi-ceph-com

$ kubectl logs pod/csi-rbdplugin-gz66z csi-rbdplugin I0515 01:19:22.469923 498118 cephcsi.go:117] Driver version: v2.1.1 and Git version: 9022d899eb6fd464f0be33701d8160ecd1317467 I0515 01:19:22.470166 498118 cephcsi.go:172] Starting driver type: rbd with name: rbd.csi.ceph.com I0515 01:19:22.472448 498118 mount_linux.go:173] Cannot run systemd-run, assuming non-systemd OS I0515 01:19:22.472470 498118 mount_linux.go:174] systemd-run failed with: exit status 1 I0515 01:19:22.472501 498118 mount_linux.go:175] systemd-run output: Failed to create bus connection: No such file or directory I0515 01:19:22.472844 498118 server.go:116] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"} I0515 01:19:23.128289 498118 utils.go:159] ID: 1 GRPC call: /csi.v1.Identity/GetPluginInfo I0515 01:19:23.128319 498118 utils.go:160] ID: 1 GRPC request: {} I0515 01:19:23.129956 498118 identityserver-default.go:37] ID: 1 Using default GetPluginInfo I0515 01:19:23.129975 498118 utils.go:165] ID: 1 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v2.1.1"} I0515 01:19:24.278851 498118 utils.go:159] ID: 2 GRPC call: /csi.v1.Node/NodeGetInfo I0515 01:19:24.278891 498118 utils.go:160] ID: 2 GRPC request: {} I0515 01:19:24.279905 498118 nodeserver-default.go:58] ID: 2 Using default NodeGetInfo I0515 01:19:24.279919 498118 utils.go:165] ID: 2 GRPC response: {"accessible_topology":{},"node_id":"selkie"} I0515 01:20:22.757173 498118 utils.go:159] ID: 3 GRPC call: /csi.v1.Identity/Probe I0515 01:20:22.757196 498118 utils.go:160] ID: 3 GRPC request: {} I0515 01:20:22.757820 498118 utils.go:165] ID: 3 GRPC response: {} I0515 01:21:22.757246 498118 utils.go:159] ID: 4 GRPC call: /csi.v1.Identity/Probe I0515 01:21:22.757287 498118 utils.go:160] ID: 4 GRPC request: {} I0515 01:21:22.758230 498118 utils.go:165] ID: 4 GRPC response: {} I0515 01:21:40.776697 498118 utils.go:159] ID: 5 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:21:40.776725 498118 utils.go:160] ID: 5 GRPC request: {} I0515 01:21:40.777603 498118 utils.go:165] ID: 5 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:21:40.786044 498118 utils.go:159] ID: 6 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:21:40.786077 498118 utils.go:160] ID: 6 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:21:40.788302 498118 utils.go:163] ID: 6 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:21:41.378927 498118 utils.go:159] ID: 7 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:21:41.378960 498118 utils.go:160] ID: 7 GRPC request: {} I0515 01:21:41.381343 498118 utils.go:165] ID: 7 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:21:41.389367 498118 utils.go:159] ID: 8 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:21:41.389389 498118 utils.go:160] ID: 8 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:21:41.391536 498118 utils.go:163] ID: 8 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:21:42.482413 498118 utils.go:159] ID: 9 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:21:42.482437 498118 utils.go:160] ID: 9 GRPC request: {} I0515 01:21:42.483292 498118 utils.go:165] ID: 9 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:21:42.493045 498118 utils.go:159] ID: 10 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:21:42.493074 498118 utils.go:160] ID: 10 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:21:42.495065 498118 utils.go:163] ID: 10 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:21:44.590650 498118 utils.go:159] ID: 11 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:21:44.590679 498118 utils.go:160] ID: 11 GRPC request: {} I0515 01:21:44.591659 498118 utils.go:165] ID: 11 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:21:44.600713 498118 utils.go:159] ID: 12 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:21:44.600737 498118 utils.go:160] ID: 12 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:21:44.602790 498118 utils.go:163] ID: 12 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:21:48.704836 498118 utils.go:159] ID: 13 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:21:48.704861 498118 utils.go:160] ID: 13 GRPC request: {} I0515 01:21:48.705801 498118 utils.go:165] ID: 13 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:21:48.715018 498118 utils.go:159] ID: 14 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:21:48.715043 498118 utils.go:160] ID: 14 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:21:48.717166 498118 utils.go:163] ID: 14 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:21:56.733622 498118 utils.go:159] ID: 15 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:21:56.733645 498118 utils.go:160] ID: 15 GRPC request: {} I0515 01:21:56.734507 498118 utils.go:165] ID: 15 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:21:56.743796 498118 utils.go:159] ID: 16 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:21:56.743819 498118 utils.go:160] ID: 16 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:21:56.746050 498118 utils.go:163] ID: 16 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:22:12.789667 498118 utils.go:159] ID: 17 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:22:12.789691 498118 utils.go:160] ID: 17 GRPC request: {} I0515 01:22:12.790651 498118 utils.go:165] ID: 17 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:22:12.799634 498118 utils.go:159] ID: 18 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:22:12.799665 498118 utils.go:160] ID: 18 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:22:12.801906 498118 utils.go:163] ID: 18 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:22:22.757227 498118 utils.go:159] ID: 19 GRPC call: /csi.v1.Identity/Probe I0515 01:22:22.757256 498118 utils.go:160] ID: 19 GRPC request: {} I0515 01:22:22.758098 498118 utils.go:165] ID: 19 GRPC response: {} I0515 01:22:44.901559 498118 utils.go:159] ID: 20 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:22:44.901582 498118 utils.go:160] ID: 20 GRPC request: {} I0515 01:22:44.902481 498118 utils.go:165] ID: 20 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:22:44.911831 498118 utils.go:159] ID: 21 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:22:44.911850 498118 utils.go:160] ID: 21 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:22:44.913715 498118 utils.go:163] ID: 21 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:23:22.757184 498118 utils.go:159] ID: 22 GRPC call: /csi.v1.Identity/Probe I0515 01:23:22.757212 498118 utils.go:160] ID: 22 GRPC request: {} I0515 01:23:22.758072 498118 utils.go:165] ID: 22 GRPC response: {} I0515 01:23:48.923585 498118 utils.go:159] ID: 23 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:23:48.923611 498118 utils.go:160] ID: 23 GRPC request: {} I0515 01:23:48.924499 498118 utils.go:165] ID: 23 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:23:48.933496 498118 utils.go:159] ID: 24 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:23:48.933518 498118 utils.go:160] ID: 24 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:23:48.935561 498118 utils.go:163] ID: 24 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node I0515 01:24:22.757194 498118 utils.go:159] ID: 25 GRPC call: /csi.v1.Identity/Probe I0515 01:24:22.757220 498118 utils.go:160] ID: 25 GRPC request: {} I0515 01:24:22.758089 498118 utils.go:165] ID: 25 GRPC response: {} I0515 01:25:22.757191 498118 utils.go:159] ID: 26 GRPC call: /csi.v1.Identity/Probe I0515 01:25:22.757220 498118 utils.go:160] ID: 26 GRPC request: {} I0515 01:25:22.758158 498118 utils.go:165] ID: 26 GRPC response: {} I0515 01:25:50.953637 498118 utils.go:159] ID: 27 GRPC call: /csi.v1.Node/NodeGetCapabilities I0515 01:25:50.953664 498118 utils.go:160] ID: 27 GRPC request: {} I0515 01:25:50.954481 498118 utils.go:165] ID: 27 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}}]} I0515 01:25:50.962507 498118 utils.go:159] ID: 28 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC call: /csi.v1.Node/NodeStageVolume I0515 01:25:50.962528 498118 utils.go:160] ID: 28 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC request: {"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:25:50.964644 498118 utils.go:163] ID: 28 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node

If the issue is in PVC mounting please attach complete logs of below containers.

Additional context

I suspect this is related to microk8s path modifications, or permissions. All of the processes run as root, and the sockets are seen in the file paths as expected. The path /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv is created by the PVC mount and global folder never appear.

nixpanic commented 4 years ago

The error comes from util/validate.go which uses the staging path it received from the request. The StagingTargetPath is passed from the Kubernetes-CSI layer into the cephcsi NodeServer. Unfortunately cephcsi does not seem to log the staging path in error cases.

You might be able to increase the verbose/debug level of the side-car container on the node where the volume is to be mounted. Maybe that will log the full request and show the staging path. With that, you might be able to identify the problem.

Madhu-1 commented 4 years ago

{"secrets":"stripped","staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs","mount_flags":["discard"]}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"28c76192-9586-11ea-9b35-00163e408a9d","imageFeatures":"layering","journalPool":"xfs-pool","pool":"xfs-pool","storage.kubernetes.io/csiProvisionerIdentity":"1589503606225-8081-rbd.csi.ceph.com"},"volume_id":"0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff"} E0515 01:25:50.964644 498118 utils.go:163] ID: 28 Req-ID: 0001-0024-28c76192-9586-11ea-9b35-00163e408a9d-0000000000000003-9bb25594-9645-11ea-a90d-1ef02b6b9cff GRPC error: rpc error: code = InvalidArgument desc = staging path does not exists on node

the staging path is available in the request, let me give a try to deploy cephcsi in microk8s I tried last time with the rook. with few modifications, it was working

nixpanic commented 4 years ago

the staging path is available in the request

Ah, yeah, it gets logged already!

let me give a try to deploy cephcsi in microk8s I tried last time with the rook. with few modifications, it was working

Maybe we should try to run e2e tests on microk8s in addition to a minikube environment?

Madhu-1 commented 4 years ago

the staging path is available in the request

Ah, yeah, it gets logged already!

let me give a try to deploy cephcsi in microk8s I tried last time with the rook. with few modifications, it was working

Maybe we should try to run e2e tests on microk8s in addition to a minikube environment?

it will be a more work, we can do it.

Madhu-1 commented 4 years ago

volumeMounts:

  • name: socket-dir mountPath: /csi
  • mountPath: /dev name: host-dev
  • mountPath: /sys name: host-sys
  • mountPath: /run/mount name: host-mount
  • mountPath: /lib/modules name: lib-modules readOnly: true
  • name: ceph-csi-config mountPath: /etc/ceph-csi-config/
  • name: ceph-csi-encryption-kms-config mountPath: /etc/ceph-csi-encryption-kms-config/
  • name: plugin-dir mountPath: /var/lib/kubelet/plugins mountPropagation: "Bidirectional"
  • name: mountpoint-dir mountPath: /var/lib/kubelet/pods mountPropagation: "Bidirectional"
  • name: keys-tmp-dir mountPath: /tmp/csi/keys

@chris-sanders can you update the volume mount path of kubelet also and see it fixes the issue

Madhu-1 commented 4 years ago

host path /var/snap/microk8s/common/var/lib/kubelet/plugins is mapped to container /var/lib/kubelet as the /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-14670fe3-b4c6-4924-8660-dcbcfdc05504/globalmount staging path directory is not present on the container it will throw the error, if you map the directories to the same path as host path it should work.

chris-sanders commented 4 years ago

You're right that was it, pod started right up after that. I didn't even catch that the path was the full hostPath. I was thinking all of the modified host paths were mapped to their standard locations in the containers and the rest of the plugin wouldn't know the difference.

For future travelers these two edits fixed it:

        - name: plugin-dir
          mountPath: /var/snap/microk8s/common/var/lib/kubelet/plugins
          # /var/lib/kubelet/plugins
          mountPropagation: "Bidirectional"
        - name: mountpoint-dir
          mountPath: /var/snap/microk8s/common/var/lib/kubelet/pods 
          # /var/lib/kubelet/pods
          mountPropagation: "Bidirectional"
Berndinox commented 3 years ago

@chris-sanders i do run into the same issue, do use microk8s 1.19.3 where do i have to set those vars?

i just can find the Var here: https://github.com/rook/rook/tree/c953de538bef894f8cba1dc3dad132d2ab66c99e/cluster/examples/kubernetes/ceph/csi/template/cephfs

However, i followed the docs to setup a ceph cluster and cephFS: https://rook.io/docs/rook/v1.5/ceph-filesystem.html -> without HELM

I basicly did a symlink from /var/lib/kubelet to /var/snap/microk8s/common/var/lib/kubelet/ Everything looks fine, but not the mount...

Unable to attach or mount volumes: unmounted volumes=[image-store], unattached volumes=[image-store default-token-pjgnd]: timed out waiting for the condition
kubelet k8s01

MountVolume.MountDevice failed for volume "pvc-d8756fb2-452d-4455-ab9e-038e30c119c5" : rpc error: code = InvalidArgument desc = staging path /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d8756fb2-452d-4455-ab9e-038e30c119c5/globalmount does not exists on node
kubelet k8s01
chris-sanders commented 3 years ago

@Berndinox the valies I've been using were removed and the chart updated to allow you to set the kubelet directory here: https://github.com/ceph/ceph-csi/pull/1736

I haven't had a chance to test the update, but hopefully that exposes the values you need in the chart. Otherwise, there was no chart value you allow you to set them. I was using ytt to patch the final yaml manifests to make the changes.

NewMountain commented 2 years ago

For anyone that finds themself here after finding this error running Microk8s and using the 1.8 Rook/Ceph Operator, here's the fix I found. Clear the raw partitions. Uninstall everything related to Rook. Open up the operator.yaml.

cp rook/deploy/examples/operator.yaml rook/deploy/examples/microk8s-operator.yaml
# `vim rook/deploy/examples/microk8s-operator.yaml`
# Search ROOK_CSI_KUBELET_DIR_PATH
# It should originally be commented out and be ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet"
# Uncomment and set line to `   ROOK_CSI_KUBELET_DIR_PATH: "/var/snap/microk8s/common/var/lib/kubelet"`

Apply your microk8s-operator.yaml instead of the original operator and use all other steps as before.

kamiwa commented 2 years ago

@NewMountain : Thank you so much man! You've made my day!

blackliner commented 2 years ago

Small addition: if you are into GitOps and use the rook operator helm chart, what you want to change is called csi.kubeletDirPath: before (it was commented out):

csi:
#  kubeletDirPath: /var/lib/kubelet

after:

csi:
  kubeletDirPath: /var/snap/microk8s/common/var/lib/kubelet
everestx commented 1 year ago

For future travelers these two edits fixed it:

Thanks a million @chris-sanders