search

openebs / mayastor

Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.
Apache License 2.0
740 stars 106 forks source link

failed to mount volume #1033

Closed weskiller closed 2 years ago

weskiller commented 3 years ago

Follow the steps in this document https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor, Deploy a test pod, the pod is not work, 

# kubectl describe pod fio
Name:         fio
Namespace:    default
Priority:     0
Node:         main25/192.168.20.25
Start Time:   Mon, 11 Oct 2021 20:47:32 +0800
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:           
IPs:          <none>
Containers:
  fio:
    Container ID:  
    Image:         mayadata/fio
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      sleep
      1000000
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-84785 (ro)
      /volume from ms-volume (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ms-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ms-volume-claim
    ReadOnly:   false
  kube-api-access-84785:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              openebs.io/engine=mayastor
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                     From     Message
  ----     ------       ----                    ----     -------
  Warning  FailedMount  17m (x863 over 42h)     kubelet  Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[ms-volume kube-api-access-84785]: timed out waiting for the condition
  Warning  FailedMount  8m43s (x1244 over 42h)  kubelet  MountVolume.MountDevice failed for volume "pvc-3cc04f16-98f2-423d-ab20-a7fbe17e974c" : rpc error: code = Internal desc = Failed to stage volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c: attach failed: connect failed: NVMe connect failed: /dev/nvme-fabrics, No such file or directory (os error 2)
  Warning  FailedMount  3m35s (x247 over 41h)   kubelet  Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[kube-api-access-84785 ms-volume]: timed out waiting for the condition

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-3cc04f16-98f2-423d-ab20-a7fbe17e974c   1Gi        RWO            Delete           Bound    default/ms-volume-claim   mayastor-1              42h

# kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ms-volume-claim   Bound    pvc-3cc04f16-98f2-423d-ab20-a7fbe17e974c   1Gi        RWO            mayastor-1     42h

# kubectl get sc
NAME         PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
mayastor-1   io.openebs.csi-mayastor   Delete          WaitForFirstConsumer   false                  42h

# kubectl -n mayastor get msp
NAME              NODE     STATE    AGE
main25-ssd-1.6t   main25   online   42h

# kubectl -n mayastor describe msp main25-ssd-1.6t
Name:         main25-ssd-1.6t
Namespace:    mayastor
Labels:       <none>
Annotations:  <none>
API Version:  openebs.io/v1alpha1
Kind:         MayastorPool
Metadata:
  Creation Timestamp:  2021-10-11T12:39:43Z
  Finalizers:
    finalizer.mayastor.openebs.io
  Generation:  1
  Managed Fields:
    API Version:  openebs.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:disks:
        f:node:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-10-11T12:39:43Z
    API Version:  openebs.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:capacity:
        f:disks:
        f:reason:
        f:spec:
          .:
          f:disks:
          f:node:
        f:state:
        f:used:
    Manager:      unknown
    Operation:    Update
    Subresource:  status
    Time:         2021-10-11T12:39:45Z
    API Version:  openebs.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizer.mayastor.openebs.io":
    Manager:         unknown
    Operation:       Update
    Time:            2021-10-11T12:47:31Z
  Resource Version:  30452
  UID:               a5d2a16f-1cd4-4e9a-86ce-632090010f4d
Spec:
  Disks:
    /dev/sdb
  Node:  main25
Status:
  Capacity:  1917438656512
  Disks:
    aio:///dev/sdb?uuid=5831cb7b-8d48-4488-b6cb-cb8b857d7a15
  Reason:  
  Spec:
    Disks:
      /dev/sdb
    Node:  main25
  State:   online
  Used:    1073741824
Events:    <none>

# kubectl -n mayastor get msn
NAME     STATE    AGE
main25   online   42h

# kubectl -n mayastor get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
mayastor-csi-f5vsb     2/2     Running   0          42h   192.168.20.12   main12   <none>           <none>
mayastor-csi-nmrlh     2/2     Running   0          42h   192.168.20.37   main37   <none>           <none>
mayastor-csi-p7qxl     2/2     Running   0          42h   192.168.20.25   main25   <none>           <none>
mayastor-csi-tlm72     2/2     Running   0          42h   192.168.20.22   main22   <none>           <none>
mayastor-csi-vv8z5     2/2     Running   0          42h   192.168.20.13   main13   <none>           <none>
mayastor-etcd-0        1/1     Running   0          42h   10.0.7.47       main37   <none>           <none>
mayastor-z5knw         1/1     Running   0          42h   192.168.20.25   main25   <none>           <none>
moac-9589d8cb6-l9z6x   3/3     Running   0          42h   10.0.7.48       main37   <none>           <none>
nats-0                 2/2     Running   0          42h   10.0.6.4        main25   <none>           <none>

# kubectl -n mayastor logs mayastor-z5knw
[2021-10-11T12:39:45.204347620+00:00  INFO mayastor::subsys::config::pool:pool.rs:87] pool configuration saved to /var/local/mayastor/pools.yaml
[2021-10-11T12:47:31.399403058+00:00  INFO mayastor::lvs::lvs_pool:lvs_pool.rs:558] created main25-ssd-1.6t/3cc04f16-98f2-423d-ab20-a7fbe17e974c
[2021-10-11T12:47:35.141886424+00:00  INFO mayastor::bdev::nexus::nexus_bdev:nexus_bdev.rs:507] UUID set to 3cc04f16-98f2-423d-ab20-a7fbe17e974c for nexus nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c
[2021-10-11T12:47:35.141980995+00:00  INFO mayastor::bdev::nexus::nexus_bdev_children:nexus_bdev_children.rs:455] nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c: opened child bdev:///3cc04f16-98f2-423d-ab20-a7fbe17e974c?uuid=9d232436-6efe-4fc2-a6a1-68174067ffaf
[2021-10-11T12:47:35.142585017+00:00  INFO mayastor::bdev::nexus::nexus_label:nexus_label.rs:1202] creating new label for child bdev:///3cc04f16-98f2-423d-ab20-a7fbe17e974c?uuid=9d232436-6efe-4fc2-a6a1-68174067ffaf
[2021-10-11T12:47:35.142652909+00:00  INFO mayastor::bdev::nexus::nexus_label:nexus_label.rs:1308] writing label to child bdev:///3cc04f16-98f2-423d-ab20-a7fbe17e974c?uuid=9d232436-6efe-4fc2-a6a1-68174067ffaf
[2021-10-11T12:47:35.313165308+00:00  INFO mayastor::persistent_store:persistent_store.rs:112] Putting key 3cc04f16-98f2-423d-ab20-a7fbe17e974c, value {"children":[{"healthy":true,"uuid":"9d232436-6efe-4fc2-a6a1-68174067ffaf"}],"clean_shutdown":false} in store.
[2021-10-11T12:47:35.315296777+00:00  INFO mayastor::persistent_store:persistent_store.rs:122] Successfully put key 3cc04f16-98f2-423d-ab20-a7fbe17e974c, value {"children":[{"healthy":true,"uuid":"9d232436-6efe-4fc2-a6a1-68174067ffaf"}],"clean_shutdown":false} in store.
[2021-10-11T12:47:35.315371511+00:00  INFO mayastor::grpc::mayastor_grpc:mayastor_grpc.rs:459] Created nexus 3cc04f16-98f2-423d-ab20-a7fbe17e974c
[2021-10-11T12:47:35.340394537+00:00  INFO mayastor::subsys::nvmf::subsystem:subsystem.rs:222] added NS ID 1
[2021-10-11T12:47:35.340627273+00:00  INFO mayastor::subsys::nvmf::subsystem:subsystem.rs:371] started "nqn.2019-05.io.openebs:nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c"
[2021-10-11T12:47:35.340671220+00:00  INFO mayastor::grpc::mayastor_grpc:mayastor_grpc.rs:611] Published nexus 3cc04f16-98f2-423d-ab20-a7fbe17e974c under nvmf://192.168.20.25:8420/nqn.2019-05.io.openebs:nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c

# kubectl -n mayastor logs -f mayastor-csi-p7qxl mayastor-csi
[2021-10-13T06:46:54Z DEBUG mayastor_csi::node] Volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c has URI nvmf://192.168.20.25:8420/nqn.2019-05.io.openebs:nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c 
[2021-10-13T06:46:54Z DEBUG mayastor_csi::node] Attaching volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c 
[2021-10-13T06:46:54Z ERROR mayastor_csi::node] Failed to stage volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c: attach failed: connect failed: NVMe connect failed: /dev/nvme-fabrics, No such file or directory (os error 2) 
[2021-10-13T06:48:56Z DEBUG mayastor_csi::node] NodeGetCapabilities request: [StageUnstageVolume] 
[2021-10-13T06:48:56Z DEBUG mayastor_csi::node] NodeGetCapabilities request: [StageUnstageVolume] 
[2021-10-13T06:48:56Z DEBUG mayastor_csi::node] Volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c has URI nvmf://192.168.20.25:8420/nqn.2019-05.io.openebs:nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c 
[2021-10-13T06:48:56Z DEBUG mayastor_csi::node] Attaching volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c 
[2021-10-13T06:48:56Z ERROR mayastor_csi::node] Failed to stage volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c: attach failed: connect failed: NVMe connect failed: /dev/nvme-fabrics, No such file or directory (os error 2) 
[2021-10-13T06:50:58Z DEBUG mayastor_csi::node] NodeGetCapabilities request: [StageUnstageVolume] 
[2021-10-13T06:50:58Z DEBUG mayastor_csi::node] NodeGetCapabilities request: [StageUnstageVolume] 
[2021-10-13T06:50:58Z DEBUG mayastor_csi::node] Volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c has URI nvmf://192.168.20.25:8420/nqn.2019-05.io.openebs:nexus-3cc04f16-98f2-423d-ab20-a7fbe17e974c 
[2021-10-13T06:50:58Z DEBUG mayastor_csi::node] Attaching volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c 
[2021-10-13T06:50:58Z ERROR mayastor_csi::node] Failed to stage volume 3cc04f16-98f2-423d-ab20-a7fbe17e974c: attach failed: connect failed: NVMe connect failed: /dev/nvme-fabrics, No such file or directory (os error 2)

Main25:~# fdisk -l

Disk /dev/sdb: 1.76 TiB, 1919313510400 bytes, 3748659200 sectors
Disk model: SMC2208         
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

is something wrong deployment?

tiagolobocastro commented 3 years ago

At a quick glance, it seems the nvme_tcp module is not enabled on the node where fio is running

weskiller commented 2 years ago

At a quick glance, it seems the nvme_tcp module is not enabled on the node where fio is running

thanks