storage export pods not created

[papanito]

I followed the guidelines but somehow the "storage export pods" are not created. This is my storage config which uses /dev/md3.

apiVersion: kadalu-operator.storage/v1alpha1
kind: KadaluStorage
metadata:
 # This will be used as name of PV Hosting Volume
  name: storage-pool-1
spec:
  type: Replica1
  storage:
    - node: node1
      device: /dev/md3
    #- node: node2
    #  device: /dev/md3

but no pods

# kubectl get pods -nkadalu  
NAME                        READY   STATUS    RESTARTS   AGE
csi-nodeplugin-t4drf        3/3     Running   0          14h
csi-nodeplugin-xbvdq        3/3     Running   0          14h
csi-provisioner-0           4/4     Running   0          14h
operator-68649f4bb6-wz5ph   1/1     Running   0          14h

pvc thus stay in state pending

# kubectl apply -f sample-pvc.yml 
persistentvolumeclaim/pv1 created
# kubectl get pvc
NAME   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pv1    Pending                                      kadalu.replica1   12m

Operator logs

I've first applied a wrong config - 2 nodes configured instead 1. However I corrected it and re-applied again. There is nothing else logged after that

[2020-01-04 20:22:53,218] INFO [main - 405:deploy_csi_pods] - Deployed CSI Pods  manifest=/kadalu/templates/csi.yaml
[2020-01-04 20:22:53,412] DEBUG [rest - 219:request] - response body: {"kind":"StorageClassList","apiVersion":"storage.k8s.io/v1","metadata":{"selfLink":"/apis/storage.k8s.io/v1/storageclasses","resourceVersion":"436497"},"items":[]}
[2020-01-04 20:22:54,538] INFO [main - 445:deploy_storage_class] - Deployed StorageClass     manifest=/kadalu/templates/storageclass.yaml
[2020-01-05 10:11:14,330] DEBUG [main - 368:crd_watch] - Event   operation=ADDED object={'apiVersion': 'kadalu-operator.storage/v1alpha1', 'kind': 'KadaluStorage', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"kadalu-operator.storage/v1alpha1","kind":"KadaluStorage","metadata":{"annotations":{},"name":"storage-pool-1","namespace":"default"},"spec":{"storage":[{"device":"/dev/md3","node":"node1"},{"device":"/dev/md3","node":"5.9.87.14"}],"type":"Replica1"}}\n'}, 'creationTimestamp': '2020-01-05T10:11:14Z', 'generation': 1, 'name': 'stor5.9.87.14age-pool-1', 'namespace': 'default', 'resourceVersion': '560337', 'selfLink': '/apis/kadalu-operator.storage/v1alpha1/namespaces/default/kadalustorages/storage-pool-1', 'uid': '78ccca1e-a4a4-4b2c-a553-15399b7cbdec'}, 'spec': {'storage': [{'device': '/dev/md3', 'node': 'node1'}, {'device': '/dev/md3', 'node': 'node2'}], 'type': 'Replica1'}}
[2020-01-05 10:11:14,330] ERROR [main - 109:validate_volume_request] - Invalid number of storage directories/devices specified
[2020-01-05 10:11:14,331] DEBUG [main - 275:handle_added] - validation of volume request failed  yaml={'apiVersion': 'kadalu-operator.storage/v1alpha1', 'kind': 'KadaluStorage', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"kadalu-operator.storage/v1alpha1","kind":"KadaluStorage","metadata":{"annotations":{},"name":"storage-pool-1","namespace":"default"},"spec":{"storage":[{"device":"/dev/md3","node":"node1"},{"device":"/dev/md3","node":"node2"}],"type":"Replica1"}}\n'}, 'creationTimestamp': '2020-01-05T10:11:14Z', 'generation': 1, 'name': 'storage-pool-1', 'namespace': 'default', 'resourceVersion': '560337', 'selfLink': '/apis/kadalu-operator.storage/v1alpha1/namespaces/default/kadalustorages/storage-pool-1', 'uid': '78ccca1e-a4a4-4b2c-a553-15399b7cbdec'}, 'spec': {'storage': [{'device': '/dev/md3', 'node': 'node1'}, {'device': '/dev/md3', 'node': 'node2'}], 'type': 'Replica1'}}
[2020-01-05 10:12:53,170] DEBUG [main - 368:crd_watch] - Event   operation=MODIFIED object={'apiVersion': 'kadalu-operator.storage/v1alpha1', 'kind': 'KadaluStorage', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"kadalu-operator.storage/v1alpha1","kind":"KadaluStorage","metadata":{"annotations":{},"name":"storage-pool-1","namespace":"default"},"spec":{"storage":[{"device":"/dev/md3","node":"node1"}],"type":"Replica1"}}\n'}, 'creationTimestamp': '2020-01-05T10:11:14Z', 'generation': 2, 'name': 'storage-pool-1', 'namespace': 'default', 'resourceVersion': '560583', 'selfLink': '/apis/kadalu-operator.storage/v1alpha1/namespaces/default/kadalustorages/storage-pool-1', 'uid': '78ccca1e-a4a4-4b2c-a553-15399b7cbdec'}, 'spec': {'storage': [{'device': '/dev/md3', 'node': 'node1'}], 'type': 'Replica1'}}
[2020-01-05 10:12:53,170] WARNING [main - 332:handle_modified] - MODIFIED handle called, but not implemented

my disk configuration

#75 sudo fdisk -l
Disk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST33000650NS    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7DC8C4C7-714F-4628-89AB-A0F2995A369F

Device          Start        End    Sectors  Size Type
/dev/sda1        4096   33558527   33554432   16G Linux RAID
/dev/sda2    33558528   34607103    1048576  512M Linux RAID
/dev/sda3    34607104 2182090751 2147483648    1T Linux RAID
/dev/sda4  2182090752 5860533134 3678442383  1.7T Linux RAID
/dev/sda5        2048       4095       2048    1M BIOS boot

Partition table entries are not in disk order.

Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST33000650NS    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3EAD1041-D4F8-4465-9184-B452F51E8531

Device          Start        End    Sectors  Size Type
/dev/sdb1        4096   33558527   33554432   16G Linux RAID
/dev/sdb2    33558528   34607103    1048576  512M Linux RAID
/dev/sdb3    34607104 2182090751 2147483648    1T Linux RAID
/dev/sdb4  2182090752 5860533134 3678442383  1.7T Linux RAID
/dev/sdb5        2048       4095       2048    1M BIOS boot

Partition table entries are not in disk order.

Disk /dev/md0: 16 GiB, 17162043392 bytes, 33519616 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/md1: 511 MiB, 535822336 bytes, 1046528 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/md3: 1.7 TiB, 1883227226112 bytes, 3678178176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/md2: 1023.9 GiB, 1099376361472 bytes, 2147219456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[papanito]

Ok maybe not an issue at all, just deleted the old config and recreated it again

# kubectl delete KadaluStorage storage-pool-1
kadalustorage.kadalu-operator.storage "storage-pool-1" deleted
# kubectl apply -f storage-config.yml        
kadalustorage.kadalu-operator.storage/storage-pool-1 created

There is still another problem as the volume claim is still in state pending.

container kadalu-operator is not valid for pod server-storage-pool-1-0-node1-0

[amarts]

@papanito Glad the first issue got resolved.

But about still pending issue, what does the pod and pvc describe and logs show?

[papanito]

@amarts I tought to maybe create a separate issue. The log says

[2020-01-05 13:01:49,720] ERROR [glusterfsd - 125:create_and_mount_brick] - Failed to mount export brick     fstype=xfs device=/dev/md3 mountdir=/bricks/storage-pool-1/data error=[32] b'' b'mount: /bricks/storage-pool-1/data: wrong fs type, bad option, bad superblock on /dev/md3, missing codepage or helper program, or other error.'