search

openebs-archive / jiva-operator

Kubernetes Operator for managing Jiva Volumes via custom resource.
https://openebs.io
Apache License 2.0
46 stars 44 forks source link

Jiva volume mount into pod fails with `driver name jiva.csi.openebs.io not found in the list of registered CSI drivers` #208

Closed rgaduput closed 1 year ago

rgaduput commented 1 year ago

What steps did you take and what happened: I am trying to configure Jiva to create volumes with replication support on Kubernetes version v1.24.9. I have followed both helm chart way of installation and the operator one which is mentioned here in the user guide docs https://openebs.io/docs/3.3.x/user-guides/jiva/jiva-prerequisites All the OpenEBS components are up & running, able to create PV as well. But when the pod comes up it is unable to mount the PV into it. Below is the exception I see. Appreciate if you can assist about this.

kubelet MountVolume.MountDevice failed for volume "pvc-aa8cec7a-2e64-4522-8538-929780487241" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name jiva.csi.openebs.io not found in the list of registered CSI drivers

CSIDriver looks fine,

[root@k8s-master~]# k get csidrivers.storage.k8s.io 
NAME                  ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
jiva.csi.openebs.io   false            true             false             <unset>         false               Persistent   5m39s

What did you expect to happen: PV should be able successfully mounted in the pod.

The output of the following commands will help us better understand what's going on:

apiVersion: openebs.io/v1
kind: JivaVolume
metadata:
  annotations:
    openebs.io/volume-policy: example-jivavolumepolicy
  creationTimestamp: "2023-04-04T04:39:50Z"
  generation: 6
  labels:
    openebs.io/component: jiva-volume
    openebs.io/persistent-volume: pvc-aa8cec7a-2e64-4522-8538-929780487241
    openebs.io/persistent-volume-claim: example-jiva-csi-pvc
  name: pvc-aa8cec7a-2e64-4522-8538-929780487241
  namespace: openebs
  resourceVersion: "1272451"
  uid: 24af2b27-8267-44d4-b4f7-59189e4b926d
spec:
  accessType: mount
  capacity: 4Gi
  desiredReplicationFactor: 2
  iscsiSpec:
    iqn: iqn.2016-09.com.openebs.jiva:pvc-aa8cec7a-2e64-4522-8538-929780487241
    targetIP: 10.111.116.215
    targetPort: 3260
  mountInfo: {}
  policy:
    replica:
      resources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/notReady
        operator: Exists
      - effect: NoExecute
        key: node.cloudprovider.kubernetes.io/uninitialized
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unschedulable
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/out-of-disk
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/memory-pressure
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/disk-pressure
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/network-unavailable
        operator: Exists
    replicaSC: openebs-hostpath
    target:
      auxResources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      replicationFactor: 2
      resources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/notReady
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 0
  pv: pvc-aa8cec7a-2e64-4522-8538-929780487241
status:
  phase: Ready
  replicaCount: 2
  replicaStatus:
  - address: tcp://10.244.180.155:9502
    mode: RW
  - address: tcp://10.244.56.90:9502
    mode: RW
  status: RW
versionDetails:
  desired: 3.4.0
  status:
    current: 3.4.0
    dependentsUpgraded: true
    lastUpdateTime: null
apiVersion: openebs.io/v1
kind: JivaVolumePolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"openebs.io/v1alpha1","kind":"JivaVolumePolicy","metadata":{"annotations":{},"name":"example-jivavolumepolicy","namespace":"openebs"},"spec":{"replicaSC":"openebs-hostpath","target":{"replicationFactor":2}}}
  creationTimestamp: "2023-04-04T04:38:05Z"
  generation: 1
  name: example-jivavolumepolicy
  namespace: openebs
  resourceVersion: "1271950"
  uid: a3965e22-61b1-4286-a84e-f5b2bb42c821
spec:
  replicaSC: openebs-hostpath
  target:
    replicationFactor: 2
Defaulted container "csi-node-driver-registrar" out of: csi-node-driver-registrar, jiva-csi-plugin, liveness-probe
I0404 04:37:03.528549       1 main.go:164] Version: v2.3.0
I0404 04:37:03.528653       1 main.go:165] Running node-driver-registrar in mode=registration
I0404 04:37:03.529471       1 main.go:189] Attempting to open a gRPC connection with: "/plugin/csi.sock"
I0404 04:37:03.529538       1 connection.go:154] Connecting to unix:///plugin/csi.sock
I0404 04:37:12.845403       1 main.go:196] Calling CSI driver to discover driver name
I0404 04:37:12.845435       1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo
I0404 04:37:12.845442       1 connection.go:184] GRPC request: {}
I0404 04:37:12.849168       1 connection.go:186] GRPC response: {"name":"jiva.csi.openebs.io","vendor_version":"3.4.0"}
I0404 04:37:12.849300       1 connection.go:187] GRPC error: <nil>
I0404 04:37:12.849309       1 main.go:206] CSI driver name: "jiva.csi.openebs.io"
I0404 04:37:12.849340       1 node_register.go:52] Starting Registration Server at: /registration/jiva.csi.openebs.io-reg.sock
I0404 04:37:12.849535       1 node_register.go:61] Registration Server started at: /registration/jiva.csi.openebs.io-reg.sock
I0404 04:37:12.849677       1 node_register.go:91] Skipping healthz server because HTTP endpoint is set to: ""

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

rgaduput commented 1 year ago

Ok, finally we have found that it was an issue with kubelet directory. Chart has default one jiva.csiNode.kubeletDir:/var/lib/kubelet/ but in our cluster it something like /opt/kubelet. After overwriting it pv mount is working fine.