search

gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
875 stars 389 forks source link

No online storage devices in cluster #592

Closed kun-qian closed 5 years ago

kun-qian commented 5 years ago

[kubernetes cluster info] $ kubectl get node -o wide

NAME       STATUS    ROLES     AGE       VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
storage1   Ready     master    4d        v1.11.3   192.168.12.111   <none>        CentOS Linux 7 (Core)   3.10.0-957.12.2.el7.x86_64   docker://17.3.2
storage2   Ready     <none>    4d        v1.11.3   192.168.12.112   <none>        CentOS Linux 7 (Core)   3.10.0-957.12.2.el7.x86_64   docker://17.3.2
storage3   Ready     <none>    4d        v1.11.3   192.168.12.113   <none>        CentOS Linux 7 (Core)   3.10.0-957.12.2.el7.x86_64   docker://17.3.2
storage4   Ready     <none>    3d        v1.11.3   192.168.12.114   <none>        CentOS Linux 7 (Core)   3.10.0-957.12.2.el7.x86_64   docker://17.3.2

$kubectl get pods -o wide

NAME                     READY     STATUS    RESTARTS   AGE       IP               NODE
glusterfs-9q5zw          1/1       Running   0          3d        192.168.12.113   storage3
glusterfs-d8mv9          1/1       Running   0          3d        192.168.12.112   storage2
glusterfs-nhprr          1/1       Running   0          3d        192.168.12.114   storage4
heketi-86f98754c-bjqsq   1/1       Running   0          3d        10.40.0.2        storage3

$echo $HEKETI_CLI_SERVER

http://10.102.64.170:8080

$./heketi-cli topology info

Cluster Id: 1124a7aa6696acfe7994541d3797ed82

    File:  true
    Block: true

    Volumes:

    Name: heketidbstorage
    Size: 2
    Id: d7ac40b9307697ff9ebd4fd096d95231
    Cluster Id: 1124a7aa6696acfe7994541d3797ed82
    Mount: 192.168.12.113:heketidbstorage
    Mount Options: backup-volfile-servers=192.168.12.112,192.168.12.114
    Durability Type: replicate
    Replica: 3
    Snapshot: Disabled

        Bricks:
            Id: 337f2880c66e335fbd006e90d6bf7083
            Path: /var/lib/heketi/mounts/vg_728653238293c1d08d25e5efd1187ebf/brick_337f2880c66e335fbd006e90d6bf7083/brick
            Size (GiB): 2
            Node: 94421ce2acdb31dadc7577dfb392bbbc
            Device: 728653238293c1d08d25e5efd1187ebf

            Id: 42a354ab87e8d942526fe06c711ab7ef
            Path: /var/lib/heketi/mounts/vg_1b7127291792d5eb6e9fd5e22ccac657/brick_42a354ab87e8d942526fe06c711ab7ef/brick
            Size (GiB): 2
            Node: 88b1601e2d16f8dbd871637645db3997
            Device: 1b7127291792d5eb6e9fd5e22ccac657

            Id: a662cb81f9dbcf9c6c6461bd1b30cbb0
            Path: /var/lib/heketi/mounts/vg_0fee46b0057cdaf47ef485310175ae52/brick_a662cb81f9dbcf9c6c6461bd1b30cbb0/brick
            Size (GiB): 2
            Node: 29f25545dd21461181c2f78658113bf9
            Device: 0fee46b0057cdaf47ef485310175ae52

    Nodes:

    Node Id: 29f25545dd21461181c2f78658113bf9
    State: online
    Cluster Id: 1124a7aa6696acfe7994541d3797ed82
    Zone: 1
    Management Hostnames: storage3
    Storage Hostnames: 192.168.12.113
    Devices:
        Id:0fee46b0057cdaf47ef485310175ae52   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):2       Free (GiB):37
            Bricks:
                Id:a662cb81f9dbcf9c6c6461bd1b30cbb0   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_0fee46b0057cdaf47ef485310175ae52/brick_a662cb81f9dbcf9c6c6461bd1b30cbb0/brick

    Node Id: 88b1601e2d16f8dbd871637645db3997
    State: online
    Cluster Id: 1124a7aa6696acfe7994541d3797ed82
    Zone: 1
    Management Hostnames: storage2
    Storage Hostnames: 192.168.12.112
    Devices:
        Id:1b7127291792d5eb6e9fd5e22ccac657   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):2       Free (GiB):37
            Bricks:
                Id:42a354ab87e8d942526fe06c711ab7ef   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_1b7127291792d5eb6e9fd5e22ccac657/brick_42a354ab87e8d942526fe06c711ab7ef/brick

    Node Id: 94421ce2acdb31dadc7577dfb392bbbc
    State: online
    Cluster Id: 1124a7aa6696acfe7994541d3797ed82
    Zone: 1
    Management Hostnames: storage4
    Storage Hostnames: 192.168.12.114
    Devices:
        Id:728653238293c1d08d25e5efd1187ebf   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):2       Free (GiB):37
            Bricks:
                Id:337f2880c66e335fbd006e90d6bf7083   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_728653238293c1d08d25e5efd1187ebf/brick_337f2880c66e335fbd006e90d6bf7083/brick

$./heketi-cli volume create --size=1

Error: Failed to allocate new volume: No online storage devices in cluster

$kubectl logs heketi-86f98754c-bjqsq

[heketi] INFO 2019/05/27 08:44:34 Allocating brick set #0
[heketi] INFO 2019/05/27 08:45:04 Periodic health check status: node 88b1601e2d16f8dbd871637645db3997 up=false
[kubeexec] ERROR 2019/05/27 08:45:04 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: Get https://10.96.0.1:443/api/v1/namespaces/default/pods?labelSelector=glusterfs-node: dial tcp 10.96.0.1:443: i/o timeout
[kubeexec] ERROR 2019/05/27 08:45:04 heketi/pkg/remoteexec/kube/target.go:135:kube.TargetDaemonSet.GetTargetPod: Failed to get list of pods
[cmdexec] ERROR 2019/05/27 08:45:04 heketi/executors/cmdexec/peer.go:81:cmdexec.(*CmdExecutor).GlusterdCheck: Failed to get list of pods
[heketi] ERROR 2019/05/27 08:45:04 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:04 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:44:18Z | 500 |   46.236719689s | 192.168.12.111:32264 | POST /volumes
[cmdexec] INFO 2019/05/27 08:45:04 Check Glusterd service status in node storage4
[heketi] INFO 2019/05/27 08:45:04 Allocating brick set #0
[kubeexec] ERROR 2019/05/27 08:45:34 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: Get https://10.96.0.1:443/api/v1/namespaces/default/pods?labelSelector=glusterfs-node: dial tcp 10.96.0.1:443: i/o timeout
[kubeexec] ERROR 2019/05/27 08:45:34 heketi/pkg/remoteexec/kube/target.go:135:kube.TargetDaemonSet.GetTargetPod: Failed to get list of pods
[cmdexec] ERROR 2019/05/27 08:45:34 heketi/executors/cmdexec/peer.go:81:cmdexec.(*CmdExecutor).GlusterdCheck: Failed to get list of pods
[heketi] INFO 2019/05/27 08:45:34 Periodic health check status: node 94421ce2acdb31dadc7577dfb392bbbc up=false
[heketi] ERROR 2019/05/27 08:45:34 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:34 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:44:40Z | 500 |   54.711146924s | 10.40.0.2:8080 | POST /volumes
[heketi] INFO 2019/05/27 08:45:34 Cleaned 0 nodes from health cache
[heketi] INFO 2019/05/27 08:45:34 Allocating brick set #0
[heketi] ERROR 2019/05/27 08:45:34 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:34 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:45:18Z | 500 |   16.236502944s | 192.168.12.111:32264 | POST /volumes
[heketi] INFO 2019/05/27 08:45:34 Allocating brick set #0
[heketi] ERROR 2019/05/27 08:45:34 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:34 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:45:34Z | 500 |   607.573399ms | 10.102.64.170:8080 | POST /volumes
[heketi] INFO 2019/05/27 08:45:40 Allocating brick set #0
[heketi] ERROR 2019/05/27 08:45:40 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:40 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:45:40Z | 500 |   772.73µs | 10.40.0.2:8080 | POST /volumes
[heketi] INFO 2019/05/27 08:45:48 Allocating brick set #0
[heketi] ERROR 2019/05/27 08:45:48 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:48 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:45:48Z | 500 |   810.177µs | 192.168.12.111:32264 | POST /volumes
[heketi] INFO 2019/05/27 08:45:55 Allocating brick set #0
[negroni] 2019-05-27T08:45:55Z | 500 |   1.548224ms | 10.40.0.2:8080 | POST /volumes
[heketi] ERROR 2019/05/27 08:45:55 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:45:55 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[heketi] INFO 2019/05/27 08:46:03 Allocating brick set #0
[heketi] ERROR 2019/05/27 08:46:03 heketi/apps/glusterfs/volume_entry_allocate.go:60:glusterfs.(*VolumeEntry).allocBricksInCluster: No online storage devices in cluster
[heketi] ERROR 2019/05/27 08:46:03 heketi/apps/glusterfs/operations_manage.go:211:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No online storage devices in cluster
[negroni] 2019-05-27T08:46:03Z | 500 |   836.338µs | 192.168.12.111:32264 | POST /volumes
[heketi] INFO 2019/05/27 08:46:04 Starting Node Health Status refresh
[cmdexec] INFO 2019/05/27 08:46:04 Check Glusterd service status in node storage3
[heketi] INFO 2019/05/27 08:46:10 Allocating brick set #0

Question: The topology info shows that the cluster, devices are OK. But can't create a volume. Very confused.

kun-qian commented 5 years ago

reinstall is OK for me.