Cannot get gluster-block storage class work

[yuanlinios]

I followed this guide https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/container-native_storage_for_openshift_container_platform/block_storage to configure gluster-block in openshift origin 3.9, but could NEVER get it work.

I am using external glusterfs storage nodes, but containerized/native heketi hosted in openshift.

Version

oc v3.9.0+191fece kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift.red.example.com:8443 openshift v3.9.0+a96a520-22 kubernetes v1.9.1+a0ce1bc657

Steps To Reproduce

1. deploy an openshift origin 3.9 cluster with glusterfs and gluster-block storage class
2. create a pvc to request gluster-block volume

Current Result

the pvc is always pending

Expected Result

the pvc should be bound to a pv created dynamically

Additional Information

The glusterfs storage nodes are prepared like this

yum -y install centos-release-gluster41
yum -y install glusterfs glusterfs-fuse glusterfs-libs glusterfs-server
wipefs -a /dev/sdc --force
systemctl enable glusterd
systemctl start glusterd
systemctl status glusterd

yum install gluster-block -y
modprobe target_core_user
systemctl start gluster-blockd
systemctl enable gluster-blockd

Openshift atomic hosts are prepared as follows:

mpathconf --enable 
echo dm_multipath > /etc/modules-load.d/dm_multipath.conf

cat >> /etc/multipath.conf <<EOF
devices {
    device { 
        vendor "LIO-ORG"
        user_friendly_names "yes"       # names like mpatha
        path_grouping_policy "failover" # one path per group
        path_selector "round-robin 0"
        failback immediate
        path_checker "tur"
        prio "const"
        no_path_retry 120
        rr_weight "uniform"
    }
}
EOF

systemctl restart multipathd
systemctl status multipathd

And here is my inventory file

[OSEv3:children]
masters
glusterfs
nodes
etcd

[masters]
atomic11a.lab.local
atomic16a.lab.local
atomic17a.lab.local

[etcd]
atomic11a.lab.local
atomic16a.lab.local
atomic17a.lab.local

[glusterfs]
centos34a.lab.local glusterfs_ip='10.65.150.34' glusterfs_devices='["/dev/sdc"]'
centos35a.lab.local glusterfs_ip='10.65.150.35' glusterfs_devices='["/dev/sdc"]'
centos40a.lab.local glusterfs_ip='10.65.150.40' glusterfs_devices='["/dev/sdc"]'

[nodes]
atomic11a.lab.local openshift_node_labels="{'region': 'master'}"
atomic16a.lab.local openshift_node_labels="{'region': 'master'}"
atomic17a.lab.local openshift_node_labels="{'region': 'master'}"

atomic12a.lab.local openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
atomic18a.lab.local openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

atomic13a.lab.local openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
atomic19a.lab.local openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

[OSEv3:vars]
ansible_ssh_user=centos
ansible_become=true
debug_level=4
containerized=true
openshift_deployment_type=origin

openshift_release=v3.9
openshift_image_tag=v3.9.0
osm_etcd_image=registry.fedoraproject.org/latest/etcd:latest
openshift_storage_glusterfs_heketi_version=latest

openshift_storage_glusterfs_is_native=false
openshift_storage_glusterfs_heketi_is_native=true
openshift_storage_glusterfs_heketi_executor=ssh
openshift_storage_glusterfs_heketi_ssh_port=22
openshift_storage_glusterfs_heketi_ssh_user=centos
openshift_storage_glusterfs_heketi_ssh_sudo=true
openshift_storage_glusterfs_heketi_ssh_keyfile=/home/centos/.ssh/id_rsa

openshift_master_default_subdomain=red.example.com
openshift_master_cluster_hostname=master-red.lab.local
openshift_master_cluster_public_hostname=openshift.red.example.com

openshift_router_selector='region=infra'
openshift_registry_selector='region=infra'
osm_default_node_selector='region=primary'
openshift_master_cluster_method=native

openshift_storage_glusterfs_wipe=true
openshift_storage_glusterfs_block_deploy=true
openshift_storage_glusterfs_block_host_vol_create=true
openshift_storage_glusterfs_block_host_vol_size=20
openshift_storage_glusterfs_block_host_vol_max=5
openshift_storage_glusterfs_block_storageclass=true
openshift_storage_glusterfs_registry_block_deploy=false
openshift_storage_glusterfs_storageclass=true
openshift_master_dynamic_provisioning_enabled=true
openshift_hosted_registry_storage_kind=glusterfs
openshift_hosted_registry_storage_volume_size=10Gi

After the openshift cluster is deployed successfully, I created a pvc like this

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: adhoc-block-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: glusterfs-storage-block

But the pvc is always in pending:

oc describe pvc adhoc-block-pvc
Name:          adhoc-block-pvc
Namespace:     glusterfs
StorageClass:  glusterfs-storage-block
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=gluster.org/glusterblock
Finalizers:    []
Capacity:
Access Modes:
Events:
  Type    Reason                Age                  From                         Message
  ----    ------                ----                 ----                         -------
  Normal  ExternalProvisioning  4m (x3453 over 13h)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "gluster.org/glusterblock" or manually created by system administrator

Is there any mistake in my steps?

[jwforres]

@openshift/sig-storage

[openshift-merge-robot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[yuanlinios]

Hmm, it works now