`TlsVerify=False` and self-signed certificates

[bmeares]

Hi esdbclient team! I'm trying to connect to an EventStoreDB cluster that uses a self-signed certificate, but the client tries to verify the cert anyway.

I noticed a note about this situation in TODO.md:

Connection field 'TlsVerifyCert' - what to verify? and how?

design intention: if "false" configure client to avoid checking server certificate not sure that we can do this with the Python library, although there is a callback... there is a discussion about this on GH, with a PR that wasn't merged https://github.com/grpc/grpc/issues/15461

Here's a code snippet to demonstrate:

from esdbclient import EventStoreDBClient

with open('./eventstoredb-node.pem', 'rb') as f:
    root_cert = f.read()

uri = 'esdb://<username>:<password>@<ip0>:2113,<ip1>:2113,<ip2>:2113?nodePreference=follower&connectionname=eventstoredb-node&keepalivetimeout=10000&keepAliveInterval=10000&tls=true&tlsverifycert=false'
client = EventStoreDBClient(uri, root_certificates=root_cert)

### The following line is printed 30 times (10 for each IP?), then the traceback is raised:
# E1015 19:44:56.409109434   58689 ssl_transport_security.cc:1519]       Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/client.py", line 280, in __init__
    self._connection = self._connect()
                       ^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/client.py", line 322, in _connect
    return self._discover_preferred_node()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/client.py", line 357, in _discover_preferred_node
    raise DiscoveryFailed(msg) from last_exception
esdbclient.exceptions.DiscoveryFailed: Failed to obtain cluster info from '<IP REDACTED>:2113,<IP REDACTED>:2113,<IP REDACTED>:2113': <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:<IP REDACTED>:2113: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:<IP REDACTED>:2113: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED", grpc_status:14, created_time:"2024-10-15T19:44:56.409428937+00:00"}"
>

Can this be handled on the client side, and if so should I write a patch? I believe our other clients are Java so it should be possible.

Thank you for your hard work!

[bmeares]

Update, when I pass our chain cert as the root cert, I am able to construct the client, though subscriptions still fail to verify TLS:

import esdbclient

with open('./eventstoredb-node-chain.pem', 'rb') as f:
    chain_cert = f.read()

uri = 'esdb://<username>:<password>@<ip0>:2113,<ip1>:2113,<ip2>:2113?nodePreference=follower&connectionname=eventstoredb-node&keepalivetimeout=10000&keepAliveInterval=10000&tls=true&tlsverifycert=false'

client = esdbclient.EventStoreDBClient(uri=uri, root_certificates=chain_cert)
client.subscribe_to_all()

# E1016 15:02:45.816522975 2190851 ssl_transport_security.cc:1519]       Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
# E1016 15:02:45.816522975 2190851 ssl_transport_security.cc:1519]       Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
# E1016 15:02:45.816522975 2190851 ssl_transport_security.cc:1519]       Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/client.py", line 145, in retrygrpc_decorator
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/client.py", line 133, in autoreconnect_decorator
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/client.py", line 802, in subscribe_to_all
    return self.streams.read(
           ^^^^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/streams.py", line 1144, in read
    return CatchupSubscription(
           ^^^^^^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/streams.py", line 292, in __init__
    first_read_resp = self._get_next_read_resp()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/eventstore/lib/python3.12/site-packages/esdbclient/streams.py", line 264, in _get_next_read_resp
    raise self._handle_stream_read_rpc_error(e) from None
esdbclient.exceptions.SSLError: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:<IP REDACTED>:2113: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-16T15:02:46.161022603+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:<ip>:2113: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"}"
>

[bmeares]

Update 2: Luckily, I am able to connect to a single node using the self-signed certificate. Perhaps it's something to do with how discovery?🤔 To my knowledge, the cert is shared, so I'll need to do some more digging. For now, I'll be connecting a single node as a workaround.

[johnbywater]

Hi @bmeares,

Sorry for the delay in replying. I have been away on holidays but am back now.

I think you and Kyle are discussing this with Tony, but for the public record:

Firstly, as far as I am able to determine, it isn't possible to support TlsVerifyCert=False because in the Python grpc client there is no facility to disable certificate verification. I might have missed something, but I have spent some time today, and more time in the past, looking into this. There are has been some discussion about this in the grpc project since 2018, and this is something that is possible with other languages, but in Python it seems to me that this simply isn't possible.

Secondly, I'm guessing the reason why your call subscribe_to_all() fails is the following:

1. You are using a connection string URI with the esdb schema and more than one grpc target. This means that the client will use the given grpc targets to make a connection to a server, using one after another, in turn, until a connection can be made to a server. From this connection the "cluster info" will be obtained, which is a list of "cluster member" objects that each have an "address" attribute and a "port" attribute, and "state" attribute that indicated whether the node is a "leader" or a "follower". The client will then select a node from the "cluster info", according to the "NodePreference" option in the connection string URI. The client will then derive a new grpc target, combining the "address" and "port" fields of the selected cluster memeber. The client will then create a new gRPC connection, using that derived grpc target.

2. The "address" and "port" attributes of the "cluster member" objects included in the server "cluster info" response are determined by the server environment variables EVENTSTORE_ADVERTISE_HOST_TO_CLIENT_AS and EVENTSTORE_ADVERTISE_HTTP_PORT_TO_CLIENT_AS.

3. The "address" value can be a hostname or an IP address, but must be mentioned when generating the server certificates, appropriately as an IP address or as a DNS name. This value must also allow the client to reach the server over the TCP/IP network. That is, the address must both (a) match the certificate and (b) actually allow the client to reach the server.

4. In your case, since you can do the "discovery", I'm guessing you just need to configure each server in your cluster with EVENTSTORE_ADVERTISE_HOST_TO_CLIENT_AS as one of the IP addresses mentioned in your connection string URI, so that the cluster info correctly advertises the IP addresses of the servers.

If you have already done this, then we will need to investigate further.

Please note, you can obtain the cluster info by calling the client method read_gossip(). Configure the client with a connection string URI that uses the "esdb" schema with just one grpc-target, so that the "discovery" doesn't happen.

uri = 'esdb://<username>:<password>@<ip0>:2113'
client = esdbclient.EventStoreDBClient(uri=uri, root_certificates=chain_cert)
print(client.read_gossip())

See the docs for alternative ways of configuring the "cluster info" values. https://developers.eventstore.com/server/v22.10/networking.html#advertise-to-clients

[johnbywater]

I am able to connect to a single node using the self-signed certificate.

Have a look at the response to client.read_gossip() and see if the address values are the same as you were specifying in your URI that had multiple IP addresses? If not, then configuring the server, so that they are, might resolve the original issue.