Wrong resturl in StorageClass glusterfs-registry-block (3.9)

[lentzi90]

Description

Both storage classes have the same resturl when deploying a cluster with [glusterfs] and [glusterfs_registry].

Version

Version information in the code block below.

ansible 2.5.2
openshift-ansible-3.9.29-1-25-ga32659580

Steps To Reproduce

1. Follow this guide for the inventory file, but make the ...glusterfs_storageclass_default=True instead of ...glusterfs_registry_block_storageclass_default to allow the ansible service broker to deploy properly.
2. Deploy the cluster with prerequisites.yml and then deploy_cluster.yml
3. Check the storage classes.

Expected Results

The glusterfs-registry-block storage class should not have the same resturl parameter as glusterfs-storage.

Observed Results

The glusterfs-registry-block storage class points to the heketi-storage route instead of heketi-registry. When trying to use glusterfs-registry-block for logging/metrics storage, the heketi-storage pod will crash and go into a crash loop back off.

$ oc get storageclass
NAME                          PROVISIONER                AGE
glusterfs-registry-block      gluster.org/glusterblock   2m
glusterfs-storage (default)   kubernetes.io/glusterfs    7m

$ oc describe storageclass glusterfs-registry-block
Name:            glusterfs-registry-block
IsDefaultClass:  No
Annotations:     <none>
Provisioner:     gluster.org/glusterblock
Parameters:      chapauthenabled=true,hacount=3,restsecretname=heketi-registry-admin-secret-block,restsecretnamespace=infra-storage,resturl=http://heketi-storage-glusterfs.router.default.svc.cluster.local,restuser=admin
ReclaimPolicy:   Delete
Events:          <none>

$ oc describe storageclass glusterfs-storage
Name:            glusterfs-storage
IsDefaultClass:  Yes
Annotations:     storageclass.kubernetes.io/is-default-class=true
Provisioner:     kubernetes.io/glusterfs
Parameters:      resturl=http://heketi-storage-glusterfs.router.default.svc.cluster.local,restuser=admin,secretName=heketi-storage-admin-secret,secretNamespace=glusterfs
ReclaimPolicy:   Delete
Events:          <none>

$ oc get route --all-namespaces
NAMESPACE                          NAME               HOST/PORT                                                                    PATH      SERVICES           PORT      TERMINATION   WILDCARD
...
glusterfs                          heketi-storage     heketi-storage-glusterfs.router.default.svc.cluster.local                              heketi-storage     <all>                   None
infra-storage                      heketi-registry    heketi-registry-infra-storage.router.default.svc.cluster.local                         heketi-registry    <all>                   None

Additional Information

• OS: Ubuntu 18.04
• After changing the resturl to heketi-registry instead of heketi-storage I get this problem instead.

Inventory file:

master ansible_host=ec2-35-180-100-217.eu-west-3.compute.amazonaws.com
app1 ansible_host=ec2-35-180-34-217.eu-west-3.compute.amazonaws.com
app2 ansible_host=ec2-35-180-111-111.eu-west-3.compute.amazonaws.com
app3 ansible_host=ec2-35-180-18-179.eu-west-3.compute.amazonaws.com
gfs_reg1 ansible_host=ec2-35-180-34-245.eu-west-3.compute.amazonaws.com
gfs_reg2 ansible_host=ec2-35-180-32-223.eu-west-3.compute.amazonaws.com
gfs_reg3 ansible_host=ec2-35-180-34-234.eu-west-3.compute.amazonaws.com

[OSEv3:children]
masters
nodes
etcd
glusterfs
glusterfs_registry

[OSEv3:vars]
openshift_deployment_type=origin
openshift_release=3.9
osm_cluster_network_cidr=10.128.0.0/14
openshift_portal_net=172.30.0.0/16
osm_host_subnet_length=9
openshift_disable_check=disk_availability,memory_availability

ansible_user='centos'
ansible_become=yes

openshift_master_dynamic_provisioning_enabled=true
openshift_storage_glusterfs_block_host_vol_size=20

# Registry
openshift_registry_selector="role=infra"
openshift_hosted_registry_storage_kind=glusterfs
openshift_hosted_registry_storage_volume_size=10Gi

# GlusterFS
# -------------------------------------------------------

# App storage
openshift_storage_glusterfs_block_deploy=false

# CNS storage for OpenShift infrastructure (Logging and Metrics)
openshift_storage_glusterfs_registry_namespace=infra-storage
openshift_storage_glusterfs_registry_storageclass=false
openshift_storage_glusterfs_registry_block_deploy=true
openshift_storage_glusterfs_registry_block_host_vol_create=true
openshift_storage_glusterfs_registry_block_host_vol_size=20
openshift_storage_glusterfs_registry_block_storageclass=true

openshift_storageclass_default=false
openshift_storage_glusterfs_storageclass_default=true
openshift_storage_glusterfs_registry_block_storageclass_default=false
# ---------------------------------------------------------------------

[masters]
master

[etcd]
master

[nodes]
master openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
app1 openshift_schedulable=true openshift_node_labels="{'role': 'app'}"
app2 openshift_schedulable=true openshift_node_labels="{'role': 'app'}"
app3 openshift_schedulable=true openshift_node_labels="{'role': 'app'}"
gfs_reg1 openshift_schedulable=true openshift_node_labels="{'role': 'infra'}"
gfs_reg2 openshift_schedulable=true openshift_node_labels="{'role': 'infra'}"
gfs_reg3 openshift_schedulable=true openshift_node_labels="{'role': 'infra'}"

[glusterfs]
app1 glusterfs_devices='[ "/dev/xvdb" ]'
app2 glusterfs_devices='[ "/dev/xvdb" ]'
app3 glusterfs_devices='[ "/dev/xvdb" ]'

[glusterfs_registry]
gfs_reg1 glusterfs_devices='[ "/dev/xvdb" ]'
gfs_reg2 glusterfs_devices='[ "/dev/xvdb" ]'
gfs_reg3 glusterfs_devices='[ "/dev/xvdb" ]'

[zensonic]

I have the same problem. I have tried to edit the storage class resturl to point to heketi-registry, but I am not allowed. Can you share how you did "After changing the resturl to heketi-registry instead of heketi-storage I get this problem instead"?

Thomas

[lentzi90]

I deleted the old storage class and created a new with the correct url.

oc get sc glusterfs-storage -o yaml > storage-class.yaml

Edit storage-class.yaml and then do

oc replace -f storage-class.yaml

[zensonic]

Hmm. Didn't work

oc replace -f glusterfs-registry-block.yaml

The StorageClass "glusterfs-registry-block" is invalid: parameters: Forbidden: updates to parameters are forbidden.

[lentzi90]

Then you have to first delete the old one: oc delete sc <storage-class-name> After that you can create the new one: oc create -f glusterfs-registry-block.yaml

[lentzi90]

I don't think this is a problem any more. Closing...