Simpler listener configuration for multiple domains with TLS support

[shamsimam]

Title: Simpler listener configuration for multiple domains with TLS support

Cross-post from envoy-users group where there was little response.

Description: tl;dr What is the simplest way to provide a mapping of domains to certificates and avoid generating repeated (one that can easily be templated by the domain+cert combination) filter chain configurations in the Envoy listener config?

Details:

The example for configuring multiple domains suggests configuring multiple filter chains. For example, using the examples in this document for two custom domains we have something like in our listener configurations:

filter_chains:
- filter_chain_match:
    server_names: ["foo.example.com"]
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/path/foo/cert.pem" }
          private_key: { filename: "/path/foo/key.pem" }
  # other configuration
  ...
- filter_chain_match:
    server_names: "bar.example.com"
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/path/bar/cert.pem" }
          private_key: { filename: "/path/bar/key.pem" }
  # other configuration (repeat of what we have for foo.example.com
  ...
# repeat for every other domain

When configuring listeners for many domains (in the order of thousands) this can become quite cumbersome as there is one entry per domain name with repeated associated configuration on other options and yields a very large configuration file (in the order of MBs). In addition, anytime a domain name gets added or removed, the configuration has to be regenerated (for LDS). Separately, use of CommonTlsContext means reloading the certificate needs an Envoy hot-restart.

The Envoy documentation of Certificate Management says:

Secret Discovery Service referenced certificates. By using SDS, certificates can either be referenced as files (reloading the certs when the parent directory is moved) or through an external SDS server that can push new certificates.

We explored the use of SDS file as documented in this article:

 common_tls_context:
     alpn_protocols: ["h2", "http/1.1"]
     tls_certificate_sds_secret_configs:
         sds_config:
             path: /etc/envoy/sds.yaml

But, SDS yaml files can contain only a single entry, and attempting to configure multiple domain mapping fails with the following message:

[2021-03-16 23:40:35.661][909395][warning][config] [source/common/config/filesystem_subscription_impl.cc:43] Filesystem config update rejected: Unexpected SDS secrets length: 251

As a result, using SDS with the file-based approach means we need to generate additional files, one per domain, to support dynamic refresh of certificates on top of the filter chain listener configurations described above.

Question:

What is the better alternative for configuring listeners for multiple domains along with TLS cert configuration? Ideally, we avoid repeating filter chain configurations and generating multiple files for certificates.

Relevant feature in HAProxy:

We have previously used HAProxy for similar configurations where a common “listener” configuration can be provided for multiple domains with the per domain certificate lookup dynamically configured via an external table (crt-list option). This yields a much smaller configuration file, in the order of KBs when everything is combined.

ECDS mentioned in user group:

There was some mention of ECDS in the user-group thread, but I was unable to find an example. Reading what little I was able to find about it looks promising. Do we have an idea what the example would look like for the dummy configuration with foo.example.com and bar.example.com I have above? A link to a good reference would also be much appreciated.

Thanks in advance for your help.

[yanavlasov]

@alyssawilk @htuch @mattklein123 for any comments.

[htuch]

You could add support potentially for > 1 TLS certificate of a given type in CommonTlsContext tls_certificates, but would need to deal with the challenges around overlapping SANs etc. That would still have the problem of having to update large listeners at scale.

I think FCDS would solve the scaling problem here if we implemented it https://github.com/envoyproxy/envoy/issues/4540, but I don't think anyone is working on it actively.

CC @ggreenway @envoyproxy/tls-watchers @lambdai

[ggreenway]

This has come up in various forms over time. As noted, FCDS may solve this for you, but it hasn't been implemented. Supporting SDS for multiple certs, also a TODO, would also help your situation.

Is all other listener/filter configuration identical, and you just want to map a domain to a tls certificate? (And I presume that you can't combine many SANs into the same cert? If you could, your problem would be reduced substantially because many names can map to one cert.) If so, a handshaker that just handles providing the correct certificate might solve some of your problem.

I don't think there's a better way to do what you're doing in envoy that's already implemented.

[shamsimam]

Thanks for the responses.

Is all other listener/filter configuration identical, and you just want to map a domain to a tls certificate?

That is correct.

And I presume that you can't combine many SANs into the same cert?

That is correct, we do not want to combine multiple SANs into the same certificate.

If so, a handshaker that just handles providing the correct certificate might solve some of your problem.

Are there any existing issues around this that I can follow to track progress along these lines?

[ggreenway]

Are there any existing issues around this that I can follow to track progress along these lines?

I thought I recalled someone doing this and discussing (but not open-sourcing it), but I can't find it.

There is an extension point for custom handshakers added by @ambuc, if you're interested in implementing a solution. I'm not aware of anyone working on this right now.

[shamsimam]

@ambuc Do you mind pointing me to the extension point for custom handshakers?

[ggreenway]

You can see an example in the tests: https://github.com/envoyproxy/envoy/blob/63307f55889d35ce98a34afe7a9aaaf0277304d3/test/extensions/transport_sockets/tls/handshaker_test.cc#L179

[shamsimam]

Thank you!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[htuch]

@shamsimam do you pan on working on this one?

[shamsimam]

@shamsimam do you pan on working on this one?

@htuch I am not planning on working on this at least for the next few months.

[htuch]

Ack, I'll leave this Help Wanted as it seems a reasonable ask.

[doujiang24]

I'm interested in this feature, and I have implemented a demo of custom TLS handshake extension, that can load/set certificates/keys by SNI server name, on-demand.

I think custom TLS handshake is very powerful.

I have an idea, (like @htuch said here: https://github.com/envoyproxy/envoy/issues/16174#issuecomment-828865092) that we can make TLS certificate selector more flexible, without a custom handshake extension.

Here is my thoughts, any feedbacks welcome, Thanks :)

Solution:

introduce an optional matched_domains fields in tls_certificates(also tls_certificate_sds_secret_configs). it can be omitted, use the domains in certificate by default.

This is a demo.

      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: certs/cert-1.pem
              private_key:
                filename: certs/key-1.pem
              matched_domains:
                ["foo.com", "api.foo.com"]
            - certificate_chain:
                filename: certs/cert-2.pem
              private_key:
                filename: certs/key-2.pem
              matched_domains:
                ["*.foo.com"]

This means: foo.com and api.foo.com will use cert-1, bar.foo.com and blah.foo.com ... will use cert-2.

TODO

To reach it we need to improve ServerContextImpl::selectTlsContext, select certificate by the SNI server name.

Result:

We can avoid repeated associated configuration, with this feature.

Question

But, we can not avoid regenerating configuration (for LDS), when a domain name gets added.

Maybe, we can improve the SDS service? when a domain have no matched certificate, Envoy will query SDS with the domain(not a knowned name), then, SDS will return the certificate with a name(or not existing), then, Envoy cache the certificate and refresh it by the name.

Also, we may support ACME after this change? related issue: https://github.com/envoyproxy/envoy/issues/8372 https://github.com/envoyproxy/envoy/issues/96

Any suggestions? Thanks

[karminski]

Hi, in my case, I use docker-compose to run lots of small web site, and I deploy envoy on my machine as a load balancer.

For HTTP services, it's very easy to auto-generate config for the envoy and reload it for add a new site. But for the HTTPS service, I need to add a new IP for this machine and listen on it to deploy a new cert.

So do we have any new messages on this issue?

[yanavlasov]

The approach of using filter chain matcher to select different certificates based on SNI is now deprecated. There is new feature implemented here: https://github.com/envoyproxy/envoy/pull/22036

However from your question is not clear to me if this is the same area. I suggest creating a new Issue if this is the case.

[hbrls]

There was a related issue #8853.

My current solution is:

1. use ingress to match and route multiple domains to multiple ports
2. use multiple listeners to multiple ports, each listener has only one domain

[karminski]

Does it means HTTPS listen on non-443 port?

[hbrls]

Does it means HTTPS listen on non-443 port?

A front Gateway / ELB / ... will handle the https first. The then forwarded inside vpc requests are all http on 8081 / 8082 / 8083.

[ggreenway]

The approach of using filter chain matcher to select different certificates based on SNI is now deprecated.

This approach isn't deprecated; there's just another way to do this now. For cases where a different filter chain is desired, filter chain matching on SNI is still appropriate. For cases where only certificate-selection needs to change based on SNI but all other settings are identical, the new feature is appropriate.

[shamsimam]

A front Gateway / ELB / ... will handle the https first.

@hbrls How do you select the different TLS certificates based on SNI using this approach?

[hbrls]

A front Gateway / ELB / ... will handle the https first.

@hbrls How do you select the different TLS certificates based on SNI using this approach?

https://a.com -> ELB -> http://a.com:8081 -> envoy
https://b.com -> ELB -> http://b.com:8082 -> envoy

Something like this. Use a not-envoy to handle the https part before.

[karminski]

Yeah, it works. But I just want to use envoy as an self-host HTTPS load balancer to save cloud ELB cost.......

[ramaraochavali]

As discussed https://github.com/envoyproxy/envoy/issues/25958 - if we can enhance SDS config to supply all certs on the initial connection and later update with new certs (like how CDS and LDS work), we can use that + SNI based cert selection to solve this use case.

@ggreenway I know there has been a long discussion about enhancing SDS with multiple certs in https://github.com/envoyproxy/envoy/issues/25958 and you also mentioned the same in Supporting SDS for multiple certs, also a TODO. I think the discussion in https://github.com/envoyproxy/envoy/issues/25958 got derailed with static config need. I agree that is not some thing that SDS can be enhanced with - but can we add special resource name * for SDS which means it sends all the certs that are needed for dynamic cert selection based on SNI? Would you be open to that?

[ggreenway]

Yes, I'm certainly open to a solution where SDS can provide a variable number of certs, as determined by the control plane. Someone just needs to figure out exactly what the protocol should look like to both support this new mode and not break backwards compatibility by changing semantics of the existing mode.

I'm not an expert on these types of xDS details. @htuch could probably help with that part.

[htuch]

This can be modeled as an xDS global collection @adisuissa

[ramaraochavali]

Is glob support available today for other Xds Apis to have a look on how it works?

On Fri, 26 May, 2023, 9:01 am htuch, @.***> wrote:

This can be modeled as an xDS global collection @adisuissa https://github.com/adisuissa

— Reply to this email directly, view it on GitHub https://github.com/envoyproxy/envoy/issues/16174#issuecomment-1563765985, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDINKGLFLJNT7P3XJ53AXLXIAPZBANCNFSM43TFVUTQ . You are receiving this because you commented.Message ID: @.***>

[ramaraochavali]

Looks CDS/LDS/RDS already support this and we need to extend this to SDS?

[ggreenway]

Yes, but I don't think CDS/LDS support non-glob. So need to make it support glob without breaking backwards compatibility with existing control planes.

[adisuissa]

You can look at LEDS for a glob-collection use. CDS and LDS are supported by Envoy here (test example). SDS will need similar support.

I'm not aware of control-planes that support these features.

[ramaraochavali]

Ok. Thank you. I got the information I need.