NFS-Ganesha integration with GD2

[atinmu]

Placeholder github issue to track the NFS-Ganesha integration work required. Further details to be filled in.

[kalebskeithley]

As a starting point, I see four modes of operation (four use cases): 1) no NFS, 2) simple NFS using gNFS, 3) simple NFS using ganesha, 4) clustered NFS (using storhaug and pacemaker)

Note: gNFS is disabled by default when a volume is created since 3.10.

If the admin enables gNFS on a volume, GD2 starts the glusterfs NFS servers on all the volumes where gNFS is enabled. To keep things simple, if ganesha NFS or clustered NFS is enabled then gNFS may not be enabled on any volume.

If the admin enables ganesha NFS, GD2 starts nfs-ganesha (systemctl start nfs-ganesha) on the specified node. The gluster command might be gluster enable ganesha $node To keep things simple, ganesha may not be enabled if gNFS or clustered NFS is enabled on any volume. Also to keep things simple the admin will be required to write the ganesha "export" configs. We can consider having GD2 automate this in the future. The NFS-Ganesha default config exports no volumes, so this is nearly a no-op to start the ganesha.nfsd on a node. After ganesha NFS is enabled the admin must edit the ganesha config file on the specified node to add the export. The admin should use the ganesha dbus CLI to export the volume(s). It is possible to run multiple ganesha.nfsds, but they must be on different nodes. In the case of multiple ganeshas, no ganesha.nfsd should export a volume that is exported by any other ganesha.nfsd.

If the admin enables clustered NFS, they use the storhaug CLI to setup and start, and teardown a pacemaker-based HA cluster. The admin must edit the storhaug and ganesha config files before running storhaug CLI to setup the HA cluster. If clustered NFS is enabled then neither gNFS or ganesha NFS may be enabled. Similarly if gNFS or ganesha NFS is enabled then clustered NFS may not be enabled.

[nixpanic]

Instead of hook scripts, the suggestion has been to handle events. GD2 does not provide events yet, but dbus events could be exposed. In that case, a process can listen for events, create/update the export configuration and inform nfs-ganesha over its dbus interface.

Potentially the event handler for (re)configuring nfs-ganesha can be the same process as the one for Samba (#433)?

[aravindavk]

Do we need pre hook or event for Volume Start or Stop(Or any other event)?

Events are asynchronous, so main action will not wait for the response of hooks scripts or events consumers output. For example, Volume start command will not wait for hooks scripts or events consumers response. Volume start command will be successful even if a hook script or events consumer failed.

If we are using pre scripts currently, is the expectation to stop the action if pre script is not successful?

[thotz]

There are two assumptions to be made :

• Storhaug will be ready and available, so that all the nfs-ganesha ha/export related operations will be handled by it.
• Shared storage will be used to store ganesha related configuration files (Please note changes need here does not dependent much storhaug, it effects only the running ganesha process)

Following volume commands need to be considered

• Must one Stop -- volume needed to be unexported admin tries to stop it Start -- if volume is started again after a stop, then it needed to re-exported For above commands information need to send to running ganesha process via dbus

• Good to have Delete -- better to inform that configuration file for that volume can removed from shared storage Snapshot -- better to save the configuration file used for a volume during snapshot This information need to given storhaug , may be with help of events framework

• Not very much important set/reset -- for multihead ganesha, features.cache-invalidation is required to maintain consistency among different ganesha servers and clients. So either storhaug or gfapi stack in ganesha process atleast puts a log mentioning option got disabled.

[atinmu]

Shared storage will be used to store ganesha related configuration files

@thotz Do we need an explicit shared storage volume here? Can't we store these configurations in etcd itself which is managed by GD2 as part of its clustering mechanism?

[kalebskeithley]

There's some amount of work involved in converting from storing ganesha state in the (legacy) shared storage volume to the gd2/etcd store.

Storing files in a posix filesystem is fairly agnostic about what the storage is and who provides it. If it's all self contained in the FSAL_GLUSTER layer then I'm not too concerned about using gluster-specific or etcd-specific mechanism.

[nixpanic]

I think nfs-ganesha relies on rpc.statd for the lock recovery. The state of the locks is stored on persistent storage at /var/lib/nfs/statd and the common-ha/storhaug scripts configure that directory to be on the shared storage volume.

[thotz]

I have created a google doc, summarizing all the discussions so far. Please find it on below link https://docs.google.com/document/d/1bkpKEGJKl4HW_Id1zepaS6mkQXmSlhlvSSOVdMUj-sw/edit?usp=sharing

[prashanthpai]

Reopening on request by @thotz as #519 partially fixes this

[atinmu]

@thotz Can you please update the status on what's pending from NFS-Ganesha integration perspective. Are we all good with GD2, NFS-Ganesha and GlusterFS 4.1?

[thotz]

What does GD1 with nfs-ganesha ?

-- Setting up HA, creating/managing conf files for ganesha and volumes, exporting/unexporting volume -- Uses shared storage for saving configuration files -- Also while taking snapshot, it saves the export related conf files Most taken care by gluster commands like start, stop, snapshot, set, reset, ganesha specfic(nfs-ganesha and ganesha.enable)

NFS-Ganesha integration with GD2

Storhaug does the following

-- sets up ganesha cluster -- export/unexport the volume -- uses shared storage for storing configuration file

Current plans for integration

Approach 1 : just make storhaug will take care of everything. ................................................................................................ GD2 will handle following scenarios with help of running ganesha process/shared storage 1.) for start/stop : sends the dbus signal to local running ganesha process with help of shared storage 2.) enabling gNFS(if it is supported with GD2??) : ideally it should fail if there running ganesha process, again we use shared storage to ensure it 3.) Snapshots : saves the configuration file while taking snapshot and restores the ganesha conf whenever needed from shared storage.

Approach 2 : ..................... Retains everything like GD1 does, but GD2 calls storhaug cli's instead of calling helper scripts

Enhancements can be done (in storhaug and semantics to cli)

1.) Ganesha related -- have different modes : HA, without HA, pnfs, remote

2.) Export related -- more user friendly approach for exporting multiple subdirs for a single volume

3.) Configuration related -- creates directory for each volume, save configurataion file as directory specific -- change the dependency on share storage()

4.) exporting snapshots(like mounting snapshots via fuse)

5.) add options for managing ganesha cluster as well in GD2

6.) fetching client related details from ganesha cluster

[thotz]

@atinmu @kalebskeithley @aravindavk @kshlm @prashanthpai , can you please provide suggestions up on above approaches