HAProxy_Host has possible ergonomic issues (for large clusters)

[phlogistonjohn]

Hi, I'm following up on the Ceph issue https://tracker.ceph.com/issues/61852 which was related to the closed pr #966 I filed ceph/ceph#52410 to fix the problem but wanted to point out some downsides to our approach that might be improved by future versions of ganesha.

For a full description of the issue please review the ceph tracker link. In short: if the management IPs cephadm associates with nodes are on different subnets we would put addresses into HAProxy_Host that would not match those of haproxy's client address. After some discussion, we are solving the issue on the cephadm side in the most simple and straightforward way: put all valid addresses for all cluster nodes in HAProxy_Host.

We are currently unable to know precisely where ganesha deamons will run so we use all managed nodes, so for N nodes in a cluster that have M interfaces we get a list N * M long. On 3 node my test cluster the list looks like:

        HAProxy_Hosts = 192.168.76.200, 192.168.122.100, 192.168.76.201, 192.168.122.101, 192.168.122.102, 192.168.76.202;

(We cant't put subnets in there because we don't know if the subnet is shared with trusted ceph nodes or not)

The ergonomics are fine for a human managed config file with known inputs for our automated version it gets a bit unwieldly, but not impossible, to manage.

While we were discussing the possible solutions I wondered aloud if a future version of ganesha that supported something like mTLS could trust clients that send proxy protocol messages based on that. I don't think gansesha supports TLS today but it's thought I wanted to share. We'd also be open to other suggestions aimed at simplifying the trust relationship between managed haproxy and ganesha instances.

Thanks!

[ffilz]

When we get to TLS, we could consider allowing a bypass for TLS, but note that clients will want to use TLS, so we will need some way to indentify what services each TLS connection is allowed. So we may still end up with the same list :-)

Hmm, could there be some way to use a DNS wildcard instead of an IP address?

[phlogistonjohn]

Sure, I was specifically thinking of the case where we could use mTLS such that the ganshsa server would only trust clients (haproxy) with matching certs (and vice versa for haproxy). Since cephadm would be distributing/managing what certs get assigned to the containers, this fits our use case very well, but I realize there are uses for TLS in general outside of that for Ganesha. :-)

[ffilz]

I have followed some of the NFS over TLS, but not closely. TLS IS on our radar. but it's a resource issue. As long as the certificates are easy to match with services, yes, we can use that to match instead of an IP address.