search

gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
875 stars 389 forks source link

Unable to add device: Initializing device /dev/vdb failed (failed to check device contents): timeout #619

Open yuenandi opened 5 years ago

yuenandi commented 5 years ago

I ran into a problem when adding device /dev/vdb and The log :

Creating cluster ... ID: 915dc04fcce1c9433db4ff8bc146b46d
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node node4 ... ID: 0c366e00629c064e9a4e3f16733b3cff
Adding device /dev/vdb ... Unable to add device: Initializing device /dev/vdb failed (failed to check device contents): timeout
Creating node node5 ... ID: 0482a973e8d84c8551123824ebae9c4f
Adding device /dev/vdb ... Unable to add device: Initializing device /dev/vdb failed (failed to check device contents): timeout
Creating node node6 ... ID: 891ab7c2932221713fc1bde60e5f7f5d
Adding device /dev/vdb ... Unable to add device: Initializing device /dev/vdb failed (failed to check device contents): timeout
Creating node node7 ... Unable to create node: server did not provide a message (status 500: Internal Server Error)
Error loading the cluster topology.
Please check the failed node or device and rerun this script.

Heketi's log ：

[kubeexec] ERROR 2019/10/23 07:47:04 heketi/pkg/remoteexec/log/commandlog.go:62:log.(*CommandLogger).Timeout: Timeout on command [pvcreate -qq --metadatasize=128M --dataalignment=256K '/dev/vdb'] on [pod:glusterfs-tq9z7 c:glusterfs ns:glusterfs (from host:node4 selector:glusterfs-node)]: Err[<nil>]: Stdout []: Stderr [  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/loop0 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-0 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/loop1 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/vda1 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-1 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/vda2 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-2 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-3 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-4 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-5 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-6 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-7 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-8 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-9 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-10 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-11 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-12 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-13 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-14 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-15 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/vdb not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-16 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-18 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-19 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-20 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-21 not initialized in udev database even after waiting 10000000 microseconds.
]

The initialization of the disk depends on what I do with it. Can someone help me.

yuenandi commented 5 years ago

I changed glusterfs-daemonset. Yaml to increase volume. Problem solved.

spec:
      nodeSelector:
        storagenode: glusterfs
      hostNetwork: true
      containers:
      - image: gluster/gluster-centos:latest
        imagePullPolicy: IfNotPresent
        name: glusterfs
        env:
        # alternative for /dev volumeMount to enable access to *all* devices
        - name: HOST_DEV_DIR
          value: "/mnt/host-dev"
        # set GLUSTER_BLOCKD_STATUS_PROBE_ENABLE to "1" so the
        # readiness/liveness probe validate gluster-blockd as well
        - name: GLUSTER_BLOCKD_STATUS_PROBE_ENABLE
          value: "1"
        - name: GB_GLFS_LRU_COUNT
          value: "15"
        - name: TCMU_LOGDIR
          value: "/var/log/glusterfs/gluster-block"
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
        volumeMounts:
        - name: glusterfs-heketi
          mountPath: "/var/lib/heketi"
        - name: glusterfs-run
          mountPath: "/run"
        - name: glusterfs-lvm
          mountPath: "/run/lvm"
        - name: glusterfs-udev
          mountPath: "/run/udev"
        - name: glusterfs-etc
          mountPath: "/etc/glusterfs"
        - name: glusterfs-logs
          mountPath: "/var/log/glusterfs"
        - name: glusterfs-config
          mountPath: "/var/lib/glusterd"
        - name: glusterfs-host-dev
          mountPath: "/mnt/host-dev"
        - name: glusterfs-misc
          mountPath: "/var/lib/misc/glusterfsd"
        - name: glusterfs-block-sys-class
          mountPath: "/sys/class"
        - name: glusterfs-block-sys-module
          mountPath: "/sys/module"
        - name: glusterfs-cgroup
          mountPath: "/sys/fs/cgroup"
          readOnly: true
        - name: glusterfs-ssl
          mountPath: "/etc/ssl"
          readOnly: true
        - name: kernel-modules
          mountPath: "/lib/modules"
          readOnly: true
        securityContext:
          capabilities: {}
          privileged: true
        readinessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 50
        livenessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh liveness; else systemctl status glusterd.service; fi"
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 50
      volumes:
      - name: glusterfs-udev
        hostPath:
          path: "/run/udev"

Why did I first execute./ gk-deploy-y-n glusterfs-g topology. Json --user-key=userkey --admin-key=adminkey ,there's one node successful

Ayden-Franklin commented 4 years ago

Please verify that there is no any partition on that device. You can try lsblk or pvs to verify. You'd better run wipefs -a /dev/vdb and retry.

gxm651182644 commented 4 years ago

any update ,i have the same problem,....

yuenandi commented 4 years ago

any update ,i have the same problem,.... Give it a try and see if it works out：


- name: glusterfs-udev
mountPath: "/run/udev"

volumes: