problems connecting to multiple clusters when using TLS

[stefan-kiss]

hello,

we are trying to connect to multiple foundationdb clusters (at the same time) using a client. the setup is like this

cluster1:

• ca1:
  • peer.cert.1
  • client.cert.1

cluster2:

• ca2:
  • peer.cert.2
  • client.cert.2

The same client cannot connect to both cluster1 and cluster2 at the same time. -> it seems that the network thread is only able to take TLS settings one time and it will use the same certificate when trying to connect to second cluster.

the issue goes away if we use the same ca1 emitting CA for tls certificates of cluster2

on forums it was suggested that at least on operator side that is a known issue, however i think that as long as both TLS and multiple cluster connections are supported it is unreasonable to expect that all clusters a client will try to connect to have the same certificates and/or the same signing CA.

Another suggestion was to use the multithreaded client - however it seems that only allows multiple threads if the api version is different

Tested with api version 710, fdb server/7.1.15 (expectig this to be the same on latest), java and go clients - v7.1.15

[shamsadobe]

+1

[aravind-work]

+1

[johscheuer]

This issue is based on this forum thread:https://forums.foundationdb.org/t/an-fdb-client-connecting-to-multiple-fdb-clusters/3639 correct? I'm not opinionated what we use (forum or issue) but I think using a single platform makes the communication easier.

Another suggestion was to use the multithreaded client - however it seems that only allows multiple threads if the api version is different

This statement is wrong and the idea of the multi-threaded client is to have on thread per database object (FDB cluster). The initial sentence FoundationDB client library can start multiple worker threads for each version of client that is loaded. might be confusing and needs some improvements to make it more precise. I believe the idea of this sentence was just to note how the multi-threaded client and the multi-version client would work together.

Currently, each database object is associated with exactly one of the threads, so a user would need at least N database objects to make use of N threads.

From: https://apple.github.io/foundationdb/api-general.html#multi-threaded-client

But like I said in the forum post, I believe that currently the desired functionality doesn't exist and additional method must be exposed by the bindings to achieve what you want.

[stefan-kiss]

1. correct.
2. Sorry about using two platforms, from what i understood forum would be used to get some kind of help, while i see github issues a place to actually raise issues. After the forum thread i'm convinced there is an issue, so we can discuss more here.

i agree, the documentation is a bit confusing.

Now, i'm not picky - this issue can have a solution in an update in the docs, an update in the client or an update in my understanding - on how to use the client (well, yes i agree the last probably wouldn't warrant a githib issue). Or maybe it's a feature request. Currently i'm missing some knowledge but i'll try to test more. However at the very base i don't think the behavior i'm observing is clearly documented.

Beside that, if the single threaded client does not support changing the TLS parameters it should return an error when trying to do so, not pass trough.

The following code passes trough (without returning an error) even if the client has already been initialised once, and TLS settings are already set. One would expect if you cant change the TLS settings (or the change is ignored) an error to be returned.

    err = fdb.APIVersion(710)
    if err != nil {
        return fdb.Database{}, err
    }
    no := fdb.Options()

    err = no.SetTLSCertPath(clientCrt)
    if err != nil {
        log.Fatalf("%+v", fmt.Errorf("unable to set cert path to %s: %w", tlsClientCrt, err))
    }
    err = no.SetTLSKeyPath(clientKey)
    if err != nil {
        log.Fatalf("%+v", fmt.Errorf("unable to set private key path to %s: %w", tlsClientKey, err))
    }
    err = no.SetTLSCaPath(caCrt)
    if err != nil {
        log.Fatalf("%+v", fmt.Errorf("unable to set ca file path to %s: %w", tlsCa, err))
    }
    db = fdb.MustOpenDatabase(fdbCluster)
    return db, nil

from my perspective an ideal solution would mean:

• throw an error on TLS operations that are invalid or inefficient (like setting the certificate again after it has already been set)
• update the docs on connecting to multiple clusters / specify the limitations of single client
• expose the functionality for controlling which network thread you are using in the bindings for the multi-threaded client (that's what's missing right ? is there something else ?)
• update the documentation on how to use