joachimweyl
commented
1 year ago We're looking into what it would take to use NESE CephFS on the prod cluster for RWX volumes. It's possible we could use the existing NESE MDS servers used for other projects for NERC as well. If that has issues we will likely need to allocate some of our spare prod nodes to act as the MDS (+standbys). I'm arranging a meeting with NESE admins to discuss this.
I'm also looking into what the external Ceph configuration for ODF has for parameters around specifying an external CephFS. This might limit our options one way or another depending on ODF's external CephFS capabilities/configurables.
Motivation
We're facing a number of problems in how we provision and consume storage in our OpenShift environments.
Completion criteria
Definition of Done
Description
Policy issues
Open Data Foundation wasn't designed for the Ceph-as-a-service situation that describes our relationship with NESE. Certain components require privileged access to the Ceph cluster, which makes the NESE administrators uncomfortable (for legitimate reasons). While they have been willing to grant us access anyway, this will become a bigger problem if (a) we start creating additional clusters or (b) if we want to provide storage to third-party maintained clusters. Both of those things are on our horizon.
ODF was explicitly not designed for this use case and it is unlikely we will be able to achieve changes in the implementation (particularly in the short term).
NESE uses an unsupported (by Red Hat) version of Ceph, which can lead to support challenges when we encounter problems in our OpenShift environments.
Technical issues
NESE only provides Ceph RBD access to storage. This limits our ability to consume the storage -- we can't create ReadWriteMany (RWX) volumes, which means that volumes cannot be mounted on multiple nodes at the same time. This can complicate or prevent the deployment of certain applications on our clusters.
Ceph RBD isn't a great protocol for storage-as-a-service because Ceph often requires a high level of version consistency between clients and the Ceph cluster. We would like to see NESE provide an alternative such as iSCSI (for block storage) or NFS/CIFS (for file storage) that would allow a wider range of clients to connect (without requiring every group to maintain their own rbd-to-X proxy service).
Many applications expect access to an object storage service. It would be nice if in addition to Ceph RBD storage, NESE also provided an object storage service. The alternative is that everyone runs their own service
We have identified some performance issues with NESE storage. These have so far had the biggest impact on an internal project (our xdmod implementation), but this will become a bigger problem if end user application deployments experience similar issues. We need to better characterize the performance of NESE storage so that we can identify whether these issues are related to OpenShift configuration, network configuration, or represent real performance issues on the NESE side.
References
Changelog
2023-11-08: added ZH Epic link