rook storage class issue on ocp 4.5

[rahulagrawalpsl]

Hello, We are using the guidance here to create storage classes on fresh ocp 4.x clusters to use with IBM redis operator. https://playbook.cloudpaklab.ibm.com/configure-rook-cephfs-on-openshift-4/

I am not very sure how it works but it seems that the yaml files mentioned in the link above create different storage class depending on ocp cluster version. For example for 4.5 , it created these -

csi-cephfs (default) rook-ceph-block rook-cephfs

But on 4.3 cluster , it created these -

rook-ceph-block-internal rook-ceph-cephfs-internal rook-ceph-delete-bucket-internal

IBM redis has been working fine with rook-ceph-cephfs-internal on ocp 4.3. But none of the new sc seem to be working for IBM redis instances on ocp 4.5. Whenever i try to use any of the storage classes on 4.5 cluster , i keep getting volume mount errors - (Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[conf data rqa-rqa-redis-token-pzbrv]: timed out waiting for the condition)

Could someone kindly provide more clarity on why there are different storage classes on ocp 4.3 and 4.5 and what's the different among those which is failing volume mount for redis instances ?

[wkrapohl]

What type of clusters are you talking about? Are these fyre? How did you create the 4.3 cluster if on fyre? How did you create the 4.5 cluster if fyre? Did you use OCP+Beta? or did you use Quick Burn? or did you use the old ember methond? if on fyre.

[rahulagrawalpsl]

@wkrapohl yes, there are fyre custers. 4.3 was created directly from "create embers" --> Openshift --> OCP 4.3

4.5 was created from OCP+

[wkrapohl]

Clusters created by the embers method get cephfs installed as part of the fyre cluster install. They have their own storageclass naming. They are using rook-ceph v1.1.7 from a year ago Nov 2019.

Using the ansible csi-ceph install process on OCP+Beta clusters has its own storageclass naming. The naming convention in ansible preceeded the naming that fyre used. It originated from the tf_openshift_4_tools repository which was used on fyre clusters before fyre started installing cephfs by default with every ember install.

The ansible install will install v1.3.8 rook/ceph. I have done a number of 4.5 installs onto OCP+Beta cluster using the ansible without any problems and created PVCs.

[wkrapohl]

The ansible install process for cephfs can be found here: https://github.com/IBM/community-automation/tree/master/ansible/csi-cephfs-fyre-play

[wkrapohl]

See playbook for fyre here https://playbook.cloudpaklab.ibm.com/fyre-4/.

[wkrapohl]

Other thing to note is the ocp cluster via embers, the cephfs that fyre installs uses the vdc drive of every worker for the cephfs storage. Those drives are 500G. So for example with three workers you have 1.5T of cephfs storage.

OCP+Beta, by default creates an extra 200G vdb on every worker. The ansible csi-cephfs uses the vdb drive. So your not getting as much total storage by default (600G with three worker scenario) if you create the cluster via the fyre OCP+Beta GUI.

You can create a larger vdb drive, but you have to use the API to create the cluster described in the playbook info here: https://playbook.cloudpaklab.ibm.com/fyre-4/

[akshaychopra96]

The ansible install process for cephfs can be found here: https://github.com/IBM/community-automation/tree/master/ansible/csi-cephfs-fyre-play

@wkrapohl For installing csi-cephfs onto Fyre OCP+Beta clusters, In setting up inventory section, there is the mention in first point of copying inventory file from csi-cephfs-fyre-play directory But I am not able to find any such directory. Can you please give more details on this? Thank you.

FYI: I was not able to install ansible on the cluster. When I went for installation, it gave me the following message This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

[akshaychopra96]

Hi. I have used the following link for installing csi-cephfs onto Fyre OCP+Beta clusters https://github.com/IBM/community-automation/tree/master/ansible/csi-cephfs-fyre-play

1. I have cloned the repo from the link.
2. I have modified the inventory file with the cluster details as mentioned.
3. When I went to run the following command : ansible-playbook -i inventory csi-cephfs.yml

It gives me the following error ( image attached).

There is no modification mentioned in the cs-cephfs.yml file, and I am getting error in that file, as shown in image.

Can anyone help here? Thank you.

[rayashworth]

@akshaychopra96 you are running from INF server which is RHEL 8. The scripts have not been testing on RHEL 8 and I suspect we are not compensating for RHEL 8 distro on the python install.

[rayashworth]

Many of us do all of our work on ubuntu.

[rayashworth]

I am taking a look at the python install role now...

[rayashworth]

Actually we had already put in a fix for RHEL 8.

[rayashworth]

I am closing this. Please reopen.

NOTE: the role has now been updated to allow the install of rooks latest versions.