Collin-Moore
commented
5 years ago Update, now each node has a 40GB raw device named sdb. Modified my topology to use that device and reinstalled. Now all four nodes have the Failed to initialize IB Device message that I had above.
Two of the nodes when I ran lsblk I would get the following output:
lsblk: dm-0: failed to get device path
lsblk: dm-1: failed to get device path
lsblk: dm-0: failed to get device path
lsblk: dm-1: failed to get device path
lsblk: dm-2: failed to get device path
lsblk: dm-3: failed to get device path
lsblk: dm-4: failed to get device path
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 60G 0 disk
└─sda1 8:1 0 60G 0 part /
sdb 8:16 0 40G 0 disk
sr0 11:0 1 1024M 0 romThese two display the partitions after a reboot, not sure why.
One node did not have any partitions in the lsblk output, and the other had normal lsblk output like below:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 [2019-01-24 16:25:01.853316] W [MSGID: 106117] [glusterd-handler.c:6407:__glusterd_peer_rpc_not60G 0 disk
└─sda1 8:1 0 60G 0 part /
sdb 8:16 0 40G 0 disk
├─vg_523582f49c2350faec18aec9c70dbd7c-tp_f2d04a049a5145724b7b94bac0efd6c5_tmeta 253:0 0 12M 0 lvm
│ └─vg_523582f49c2350faec18aec9c70dbd7c-tp_f2d04a049a5145724b7b94bac0efd6c5-tpool 253:2 0 2G 0 lvm
│ ├─vg_523582f49c2350faec18aec9c70dbd7c-tp_f2d04a049a5145724b7b94bac0efd6c5 253:3 0 2G 0 lvm
│ └─vg_523582f49c2350faec18aec9c70dbd7c-brick_a517663655fb980df6c8ae55f0215a7f 253:4 0 2G 0 lvm
└─vg_523582f49c2350faec18aec9c70dbd7c-tp_f2d04a049a5145724b7b94bac0efd6c5_tdata 253:1 0 2G 0 lvm
└─vg_523582f49c2350faec18aec9c70dbd7c-tp_f2d04a049a5145724b7b94bac0efd6c5-tpool 253:2 0 2G 0 lvm
├─vg_523582f49c2350faec18aec9c70dbd7c-tp_f2d04a049a5145724b7b94bac0efd6c5 253:3 0 2G 0 lvm
└─vg_523582f49c2350faec18aec9c70dbd7c-brick_a517663655fb980df6c8ae55f0215a7f 253:4 0 2G 0 lvm
sr0 11:0 1 1024M 0 romIn light of these updates any ideas what is going on?
nixpanic
So I'm trying to setup glusterfs within our 4 node kubernetes cluster (2 master and two workers). My team is trying to automate the deployment and teardown of our cluster, so we are at the step of automating glusterfs setup and teardown. I had a working cluster that could dynamically provision volumes through hetketi before this, but we need to be able to do this over and over so I attempted to clean the vm's of glusterfs and rebuild. In our case this involves recreating our block device as well.
(As an aside our team does not have full control of our infrastructure, which is why we want an alternative to reimaging the vm's or restoring from a snapshot since we cannot start that process whenever we want. This is for an academic project so keep that in mind as you read, I have a feeling these aren't best practices.)
Before I get into details here is environment info:
I also modified the
glusterfs-daemonset.yamlfile to add the NoSchedule toleration to the PodSpec, that way we could run glusterfs on the master nodes (not a great idea but its our current strategy)The current problem I am running into is after running
gk-deployto completion and all the pods are running, I check the block devices usinglsblkand see that all but one of the two workers (it flips each time I rebuild betweenworker1andworker2) look like this:Whereas one of my worker nodes I will get this output from
lsblk:Below is the output for
vgdisplayon the good nodes:This is what the bad node looks like:
The issue appears to be that glusterd cannot mount my block device. Here are the first couple glusterd logs on the problem node:
So the next thing I should probably explain is what my clean up process looks like, because there may be something I'm leaving out.
gk-deploy --abortThese seem to get rid of all the files, volume groups, and persistent volumes
glusterfs.conffile I created in the/etc/modules-load.d/directory to loaddm_thin_pool,dm_mirror, anddm_snapshotat reboot.loop0device usingsudo losetup -d /dev/loop0glusterfs.imgfile we were using as our "block device" (yes this is hacky, we'll try to get some virtual block devices soon)Then I disable and remove the
loop0.servicewhich looks like:My main questions are as follows:
loop0device is causing this and I should just stop and get some virtual block devices?In the meantime I'll keep trying to debug this, any help is much appreciated and I will do my best to quickly reply and provide additional information as needed.