Find workaround for enforcing volume `nodeSelector` on share manager or make upstream feature PR

[lorenzo-w]

Longhorn seems to randomly pick a node to run the NFS share manager pod, potentially violating the volume's nodeSelector and running it on a badly connected node. A new feature to remedy this has already been requested upstream, but currently receives little attention.

[lorenzo-w]

Most viable workaround:

Create a helm chart with the following:

• Deployment of this nfs-ganesha server image using a Longhorn PV underneath: https://hub.docker.com/r/izdock/nfs-ganesha
• PV and PVC using this server according to this guide: https://www.linuxtechi.com/configure-nfs-persistent-volume-kubernetes/

Then use this chart to manually provision a RWX volume for every workload, which requires it.

[lorenzo-w]

Alternatives:

• Migrate entire storage infrastructure to Rook: https://rook.io/docs/rook/v1.12/Getting-Started/intro/ (most effort)
• Create a Rook PVC Storage Cluster per region / backbone on top of local Longhorn volumes: https://rook.io/docs/rook/v1.12/CRDs/Cluster/pvc-cluster/ (no per-volume affinity settings)
• Deploy an OpenEBS dynamic-nfs-provisioner per region / backbone: https://github.com/openebs/dynamic-nfs-provisioner/tree/develop/deploy/helm/charts (no per-volume affinity settings)
• Use nfs-ganesha-server-and-external-provisioner: https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/tree/31b8959668467e63eb796665d7348d3db8d893c1 (probably no per-volume affinity settings)
• Use nfs-subdir-external-provisioner + own nfs server: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner (no per-volume affinity settings)

[lorenzo-w]

Seems to be solved upstream: https://github.com/longhorn/longhorn/issues/7872#issuecomment-2136729472