TLS bumping: decrypting communications between internal and external services

[LuyaoZhong]

TLS Bumping in Envoy Design Doc

2022.10.31

PoC: https://github.com/envoyproxy/envoy/pull/23192 README and configurations are in tls_bumping subdir

---

2022.07.13 4 work items were worked out.

1. Certificate Provider framework https://github.com/envoyproxy/envoy/issues/19308 https://github.com/envoyproxy/envoy/pull/19582
2. SNI-based cert selection in tls transport socket https://github.com/envoyproxy/envoy/issues/21739 https://github.com/envoyproxy/envoy/pull/22036
3. A new network filter - BumpingFilter https://github.com/envoyproxy/envoy/issues/22581 https://github.com/envoyproxy/envoy/pull/22582

4. Certificate Provider instance - LocalMimicCertProvider https://github.com/envoyproxy/envoy/pull/23063

   2022.04.24 update

Mimicking certs only based on SNI is probably not enough, we require server real certificate and ensure to copy subject, subject alt name, extensions, knowing about the RSA key strength and many more. Original proposal was to set up client-first secure connection, to meet above requirements we need server-first secure connection.

Therefore, we expect the workflow like this:

1. downstream requires accessing some external websites like "google.com", the traffic is routed to Envoy
2. Envoy receive the CLIENT_HELLO but don't do handshake with downstream until step5
3. Envoy connects "google.com" (upstream) and get real server certificate
4. Envoy copies the subject, subject alt name, extensions, etc from real server certificate and generates mimic certificate
5. Envoy does TLS handshake with downstream using mimic certificate
6. traffic is decrypted and go through Envoy network filters, especially HCM, there are many http filters and user can also expand http filter easily with WASM to plugin in many security functions.
7. traffic is encrypted and sent to upstream.

---

Original Proposal

Title: decrypting communications between internal and external services

Description:

When Envoy works as sidecar or egress gateway in service mesh, Istio takes responsibility of certification generation and pushing the configs to Envoy via xDS. But when it works like typical proxy, the internal services on the edge might access many different external websites such as Google or Bing etc, Envoy does't provide the ability to terminate this kind of TLS traffic. For this scenario, we propose a method to let Envoy generate certs dynamically and do TLS handshake. Then if the client trusts the root ca that the certs signed from, it can access external services under the control of Envoy.

Changes (straw man)

1. introduce an API to enable this feature and configure ca crt and key for signing
2. get sni from tls inpector (we need sni to generate certs, just utilize tls inspector, probably no changes)
3. generate certs according to sni
4. set the certs to SSL object and then do handshake

Any comments are welcome.

[LuyaoZhong]

@LuyaoZhong I think you're missing a chunk of work in your proposed solution: You will need a way to delay the TLS handshake until you have the cert. This will probably involve a custom handshaker, which will have the integration points with your other code that fetches/generates the cert.

@ggreenway We can delay the TLS handshake until we have the cert with current proposal. I give more details about how it work and address your comments in bumping doc.

@ggreenway kindly ping :)

[LuyaoZhong]

@ggreenway Thanks for your comments. Please have a look at my reply in bumping doc.

[LuyaoZhong]

@ggreenway Thanks for your comments to help make the design details clear, I reply in bumping doc. Please have a look and let me know if you have other concern about this design.

[LuyaoZhong]

@ggreenway Thanks for your comments to help make the design details clear, I reply in bumping doc. Please have a look and let me know if you have other concern about this design.

@ggreenway kindly ping :)

[ggreenway]

I don't have any other comments. I think more will come out as you explore implementation.

[LuyaoZhong]

@ggreenway Thanks. We will explore implementation based on design clarified in bumping doc.

4 work items were worked out:

1. Certificate Provider framework
2. SNI-based cert selection in tls transport socket
3. A new network filter - BumpingFilter
4. Certificate Provider instance - LocalMimicCertProvider

We will create issue or PR for these work item, let's we discuss the implementation details there.

cc @mattklein123 @lambdai @soulxu Thanks for your comments.

[rohrit]

Hi @LuyaoZhong, thanks for raising this as it is an important security use case. I have a question about the proposal and the intended support - would this work with HTTP CONNECT like you describe in your presentation video (https://events.istio.io/istiocon-2022/sessions/introducing-tls-bumping/) as well?

[LuyaoZhong]

Hi @LuyaoZhong, thanks for raising this as it is an important security use case. I have a question about the proposal and the intended support - would this work with HTTP CONNECT like you describe in your presentation video (https://events.istio.io/istiocon-2022/sessions/introducing-tls-bumping/) as well?

Yes, making this work with HTTP CONNECT is one of our goals. We are going to utilize internal listener feature to handle HTTP CONNECT. The data flow looks like this: Client -> ListenerA(HCM with CONNECT termination) -> InternalListenerB(TLS Bumping) -> Upstream server.

Would you mind share your use case or scenario? We'd like to know more about requirements from real world.

[rohrit]

Forward proxy is the main use case where the user points the browser/system proxy at Envoy. One of things that is not optimal in Envoy is the need for the internal listener for the bumping TLS termination but I guess a new custom filter would be needed which can satisfy both CONNECT termination and the payload TLS termination and mimicking.

[LuyaoZhong]

Forward proxy is the main use case where the user points the browser/system proxy at Envoy. One of things that is not optimal in Envoy is the need for the internal listener for the bumping TLS termination but I guess a new custom filter would be needed which can satisfy both CONNECT termination and the payload TLS termination and mimicking.

A new custom filter was an alternative, we had https://github.com/envoyproxy/envoy/pull/19346 as PoC before, but this can only work for HTTP/1.0. To support HTTP/2 even QUIC, a complicated network filter is required, and we can predict that we need to copy a lot of logic from HCM to process http protocol, but the filter is just for handling CONNECT eventually. So we need to evaluate if it is worth a new filter.

It's ideal to make CONNECT termination and TLS bumping together, but due to the limitation of envoy framework, it is hard to implement currently. :(

[amthorn]

@LuyaoZhong do you know if this implementation will be compatible with the custom handshaker extension ?? I.E. to be able to send some of the relevant server cert information (SANs, CN, etc...) or the certificate itself to a custom handshaker extension??

[LuyaoZhong]

@amthorn Sorry for late response, I missed your comment. I think the answer is yes, just let the custom handshaker extension to interact with cert provider.

[amthorn]

@LuyaoZhong What if my custom handshaker already implemented a cert-provider interface?? Can i disable the envoy one and use the one my handshaker provides?? (We pass the SNI currently which forwards to the cert provider, but with this change we'd forward the SANs/CN into the handshaker instead of the SNI)

[LuyaoZhong]

@amthorn If you are asking if cert provider can work with custom handshaker in general, Yes, you need to implement a custom handshaker which can fetch certs from a cert provider instance. We can discuss on https://github.com/envoyproxy/envoy/issues/19308 and https://github.com/envoyproxy/envoy/pull/19582

If you are asking if tls bumping can work with custom handshaker, I think it depends on custom handshaker, in our current design we rely on envoy to loading certs and create a default handshaker to select cert based on SNI, when switching custom handshaker, we need to consider implementing similar functionalities to support tls bumping.

[amthorn]

I was asking about the latter. You answered my question, I think changes would be required to a handshaker that already does it's own SNI-based certificate lookup (It is not enabled for certificate bumping).

If i understand correctly, the two options we have are that either: 1) The TLS bumping logic could send SNI to custom handshakers to generate the certificate (But would then need to ignore any certificate providers within the TLS bumping logic since that would be provided by the handshaker) which is something that is not implemented in the current plan for TLS bumping. 2) We would need to change the custom handshaker to pull out the certificate provider logic and implement it against envoy's certificate provider interface; decoupling the handshaker from the certificate provider.

I'm not expecting a response, just stating my understanding so that you can correct it if i'm wrong.

[epk]

I built a poc with xDS and ALS which kind of does what this is supposed to do: https://github.com/epk/envoy-egress-mitm:

• Starts off as a TCP/Dynamic Forward Proxy
• Using Access Log Service, mints certificates based on the SNI
• Using xDS, creates L7 filter chains + clusters for the said SNI

There's a suite in e2e which tests it against top 500 domains.

[vermajit]

@LuyaoZhong Are you guys still pursuing this?

[BenAgai]

Hi @vermajit, @epk , I'm currently considering using Envoy as forward proxy to one of my projects. Is TLS bumping already supported or the only usable code is the POC you provided in the below link: https://github.com/epk/envoy-egress-mitm

Thanks in advance!

[chris-windscribe]

Also looking at Envoy as a forward proxy, however, it's unclear to me of TLS Bumping is supported or not.

[soulxu]

Also looking at Envoy as a forward proxy, however, it's unclear to me of TLS Bumping is supported or not.

We don't support it today.

[geraldstanje]

hi @LuyaoZhong @soulxu @amthorn @epk and team is there any progress? what is the plan? any alternatives as of may 2024?