search

CCI-MOC / ops-issues

2 stars 0 forks source link

Research into RWX support for NERC OCP 4.x #622

Open msdisme opened 2 years ago

msdisme commented 2 years ago

Tracking item for exploring this with NERC team.

naved001 commented 2 years ago

on ocp-prod

naved@Naveds-MacBook-Pro ~ % oc get storageclasses ocs-external-storagecluster-cephfs -o yaml |oc neat
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    description: Provides RWO and RWX Filesystem volumes
    storageclass.kubernetes.io/is-default-class: "false"
  labels:
    app.kubernetes.io/instance: cluster-resources-ocp-prod
  name: ocs-external-storagecluster-cephfs
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  fsName: cephfs
  pool: cephfs
provisioner: openshift-storage.cephfs.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
naved001 commented 2 years ago

on ocp-staging

naved@Naveds-MacBook-Pro ~ % oc get storageclasses ocs-external-storagecluster-cephfs -o yaml |oc neat
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    description: Provides RWO and RWX Filesystem volumes
    storageclass.kubernetes.io/is-default-class: "false"
  labels:
    app.kubernetes.io/instance: cluster-resources-ocp-staging
  name: ocs-external-storagecluster-cephfs
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  fsName: cephfs
  pool: cephfs
provisioner: openshift-storage.cephfs.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
msdisme commented 2 years ago

NESE would need to provide cephfs or nfs for this. xdmod currently working around via DB

larsks commented 2 years ago

NESE currently only provides us with RBD storage. This allows us to create RWX block volumes, or RWO filesystem volumes. For RWX filesystem support, we would need CephFS, or we would need to use something other than ODF (for example, an NFS server would allow us to provide RWX filesystem volumes).

We talked about the use of object storage (e.g., storage with an S3-compatible API). This is available in our OCS/ODF enabled clusters through the Noobaa storage service, and while this will in many cases be an acceptable (or even optimal) solution, there are situations in which we really want to provide multiple pods (on multiple nodes) access to the same filesystem.