Configure Envoy TLS accepted certificate signatures algorithms

Overview of the Issue

It's not trivial to configure Envoy accepted signatures.

I'm setting up a Nomad cluster, when using Consul Connect with a sidecar_service Envoy could not communicate with Consul.

This was initially reported as a bug because I'm dumb, I'm leaving it here so people can see how to configure Envoy accepted certificate signatures in the workaround section (though you should use RSA instead).

Reproduction Steps

Using the hashicorp Debian repository, setup Nomad+Consul using ed25519-signed certs.
Attempt to use connect { sidecar_service {} }.
Envoy fails to reach Consul gRPC with SSLV3_ALERT_HANDSHAKE_FAILURE

Workaround

Use envoy v1.26.0 so you can set signature_algorithms in tls_params, when using Nomad this is done by setting the connect.sidecar_image and connect.gateway_image metas.
Copy the entire boostrap_envoy.json template from bootstrap_tpl.go

In the template, set the client algorithms in common_tls_context with a list that overlaps what Consul uses:

          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.>
            "common_tls_context": {
+              "tls_params": {
+                "signature_algorithms": [
+                  "ed25519"
+                ]
+              },
              "validation_context": {
                "trusted_ca": {
                  "inline_string": "{{ .AgentCAPEM }}"

In every connect block, override the config.

connect {
 sidecar_service {
   proxy {
     config {
       envoy_bootstrap_json_tpl = var.envoy_bootstrap_tpl
     }
   }
 }
}

Envoy TLS configuration should probably be exposed to Consul (which would allow me to enable mTLS but that's another topic).

Consul info for both Client and Server

Client info

``` agent: check_monitors = 0 check_ttls = 1 checks = 10 services = 8 build: prerelease = revision = 5e08e229 version = 1.15.2 version_metadata = consul: acl = disabled known_servers = 3 server = false runtime: arch = amd64 cpu_count = 2 goroutines = 131 max_procs = 2 os = linux version = go1.20.1 serf_lan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 33 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 53605 members = 9 query_queue = 0 query_time = 1 ``` ``` connect { enabled = true } tls { defaults { verify_incoming = true verify_outgoing = true ca_file = "…" cert_file = "…" key_file = "…" } grpc { verify_incoming = false } internal_rpc { verify_server_hostname = true } } ```

Server info

``` agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = 5e08e229 version = 1.15.2 version_metadata = consul: acl = disabled bootstrap = false known_datacenters = 1 leader = true leader_addr = x.x.x.x:8300 server = true raft: applied_index = 603699 commit_index = 603699 fsm_pending = 0 last_contact = 0 last_log_index = 603699 last_log_term = 71 last_snapshot_index = 589960 last_snapshot_term = 71 latest_configuration = [{Suffrage:Voter ID:ca4accb7-2d7c-7a1a-7683-77ecbdc40ba9 Address:x.x.x.x:8300} {Suffrage:Voter ID:448dde0b-f98a-cf8a-d5b0-82d9aed69dd3 Address:x.x.x.x:8300} {Suffrage:Voter ID:1c78c749-75f9-e266-1ebd-bb7f0da701d1 Address:x.x.x.x:8300}] latest_configuration_index = 0 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 71 runtime: arch = amd64 cpu_count = 2 goroutines = 297 max_procs = 2 os = linux version = go1.20.1 serf_lan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 33 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 53605 members = 9 query_queue = 0 query_time = 1 ``` ``` auto_encrypt { allow_tls = true } tls { defaults { verify_incoming = true verify_outgoing = true ca_file = "…" cert_file = "…" key_file = "…" } internal_rpc { verify_server_hostname = true } https { ca_file = "…" cert_file = "…" key_file = "…" verify_incoming = false } } ```

Operating system and Environment details

Kernel: Linux 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
OS: Debian GNU/Linux 11 (bullseye)
Packages from: deb [arch=amd64] https://apt.releases.hashicorp.com bullseye main

$ consul -version
Consul v1.15.2
Revision 5e08e229
Build Date 2023-03-30T17:51:19Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

$ nomad -version
Nomad v1.5.3
BuildDate 2023-04-04T20:09:50Z
Revision 434f7a1745c6304d607562daa9a4a635def7153f

The Envoy docker image tag used in ${NOMAD_envoy_version} is v1.25.1, I'm guessing this is decided by Consul but could not find where to change it.

Log Fragments

Envoy (log_level debug)

[2023-04-28 10:49:49.835][1][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2023-04-28 10:49:49.835][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:291] trying to create new connection
[2023-04-28 10:49:49.835][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2023-04-28 10:49:49.835][1][debug][http2] [source/common/http/http2/codec_impl.cc:1586] [C5] updating connection-level initial window size to 268435456
[2023-04-28 10:49:49.835][1][debug][connection] [./source/common/network/connection_impl.h:98] [C5] current connecting state: true
[2023-04-28 10:49:49.835][1][debug][client] [source/common/http/codec_client.cc:57] [C5] connecting
[2023-04-28 10:49:49.835][1][debug][connection] [source/common/network/connection_impl.cc:941] [C5] connecting to alloc/tmp/consul_grpc.sock
[2023-04-28 10:49:49.836][1][debug][connection] [source/common/network/connection_impl.cc:688] [C5] connected
[2023-04-28 10:49:49.837][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:360] [C5] Async cert validation completed
[2023-04-28 10:49:49.837][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:234] [C5] remote address:alloc/tmp/consul_grpc.sock,TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][connection] [source/common/network/connection_impl.cc:250] [C5] closing socket: 0
[2023-04-28 10:49:49.837][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:234] [C5] remote address:alloc/tmp/consul_grpc.sock,TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][client] [source/common/http/codec_client.cc:107] [C5] disconnect. resetting 0 pending requests
[2023-04-28 10:49:49.837][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:484] [C5] client disconnected, failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][router] [source/common/router/router.cc:1278] [C0][S8602616725729965611] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][http] [source/common/http/async_client_impl.cc:123] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE'

[2023-04-28 10:49:49.837][1][debug][config] [./source/common/config/grpc_stream.h:197] DeltaAggregatedResources gRPC config stream to local_agent closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][config] [source/common/config/grpc_subscription_impl.cc:115] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed

`consul monitor --log-level trace`

2023-04-28T10:51:38.490Z [TRACE] agent: [core][Server #2] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "ServerHandshake(\"127.0.0.1:51890\") failed: tls: peer doesn't support any of the certificate's signature algorithms"

Side to side Client Hello and Server Hello:

tls

hashicorp / consul