search

ceph / ceph-nvmeof

Service to provide Ceph storage over NVMe-oF/TCP protocol
GNU Lesser General Public License v3.0
87 stars 46 forks source link

NVMe-oF Persistent reservation support #41

Open hydro-b opened 2 years ago

hydro-b commented 2 years ago

SPDK supports NVMe-oF Persistent reservations since v19.07. The state is saved in a JSON file on local filesystem. Ideally this state would be saved as an object in RADOS on Ceph. This would open the possibility to create a high-availabe ceph-nvmeof solution with Persistent reservations support. This support is currently lacking from Ceph iSCSI gateway solution. Use cases like Microsoft Failover Clusters for use with MSSQL / Hyper-V could be supported this way (in a HA fashion).

PepperJo commented 2 years ago

We could support persistent reservations (PR) for a single gateway configuration, however as soon as we deploy gateway groups this gets more complex. For now we are planning to have compare & write fuse command support to allow coordination (e.g. VMWare uses this in VMFS). Either way PR is certainly something we can put on the list of features for the future.

benhanokh commented 1 year ago

Seems to me that this needs to be implemented together with a simple reserve and ACL as a common filter on the GW before passing commands down to the OSDs. The same mechanism should be used to store the state in a Rados object and synchronized it across the cluster before returning status to the host. The RSV/PR-RSV/ACL state must be kept in the GW memory at all times to know which hosts can access which NS

benhanokh commented 1 year ago

I heard from Orit that we limit access to a single RBD-client to any RBD-Volume. If that is the case then why do we need reservation support?

PepperJo commented 1 year ago

We don't enforce a limit like this. We believe most users will use it this way but if the user chooses to access an image via multiple path we allow it.