Provide ReadWriteMany (RWX) Storage for NERC

joachimweyl commented 1 year ago

Motivation

NESE only provides Ceph RBD access to storage. This limits our ability to consume the storage -- we can't create ReadWriteMany (RWX) volumes, which means that volumes cannot be mounted on multiple nodes at the same time. This can complicate or prevent the deployment of certain applications on our clusters.

Completion Criteria

NERC has RWX storage available for all projects.

Description

[x] Track down viable RWX storage
[x] Test it
[ ] Decide if it is viable for our uses
[ ] Implement it
- [ ] Create extra issues if this is going to be a large project

Notes

Testing Weka.io as a possibility
We're looking into what it would take to use NESE CephFS on the prod cluster for RWX volumes. It's possible we could use the existing NESE MDS servers used for other projects for NERC as well. If that has issues we will likely need to allocate some of our spare prod nodes to act as the MDS (+standbys). I'm arranging a meeting with NESE admins to discuss this.
- I'm also looking into what the external Ceph configuration for ODF has for parameters around specifying an external CephFS. This might limit our options one way or another depending on ODF's external CephFS capabilities/configurables.

Completion dates

Desired - early 2024 Required - mid 2024

msdisme commented 2 months ago

@jtriley I thought this was available with latest ceph update - is the next step testing

joachimweyl commented 2 months ago

@jtriley after your testing is this viable for our usage needs?

jtriley commented 2 days ago

Just an update that we now have a NERC CephFS volume configured (along with a dedicated MDS node that we'll need to pay back if we decide to use it in production) on NESE. I'm setting up a time to meet to discuss required cephx credentials and then test out getting RWX volumes working on NESE CephFS on the ocp-test cluster.

nerc-project / operations