Support TLS client authentication with proxy

[johanstokking]

Summary

Support TLS client authentication with potential proxy support.

Why do we need this?

To authenticate clients using TLS client authentication on listeners.

Also to support layer 7 load balancing for endpoints that are secured with TLS client authentication.

What is already there? What do you see now?

• TLS listener without client authentication
• Interop only has TLS listener and its authentication checks TLS connection state
• Basic Station frontend doesn't support TLS client authentication yet

What is missing? What do you want to see?

Support for TLS client authentication;

1. Join Server gRPC (#4)
2. Basic Station (#137 and #558)

How do you propose to implement this?

Add TLS client settings to the TLS configuration, i.e. request client certificates and specify configurable client CAs

We also need some middleware;

1. For HTTP, if there's a connection coming in through an insecure listener from a TLS proxy that did client authentication, we should check the headers. There's doesn't seem to be a standard for this, so this probably has to become configurable too
2. For gRPC, is there something similar with headers to support a proxy? @htdvisser
3. For gRPC, there needs to be middleware that puts the DN in the context, through auth.NewContextWithX509DN(). This is for the JS to test against the configured addresses (#733)

The reason that TLS proxy support is explicitly part of this issue, is for forward-compatibility. Using the x509 DN from the context decouples checking the TLS connection state in places where we don't know if that component actually did TLS termination.

Can you do this yourself and submit a Pull Request?

See issues per case

[KrishnaIyer]

Issue for Basic Station: #558

[johanstokking]

Refs #137

[johanstokking]

Join Server support: #4

[htdvisser]

2. For gRPC, is there something similar with headers to support a proxy? @htdvisser

gRPC headers are handled by the metadata package. We'd need an HTTP/2 capable proxy that can inject HTTP/2 headers into the request and then we should be able to read them from the metadata in our own interceptors.

3. For gRPC, there needs to be middleware that puts the DN in the context, through auth.NewContextWithX509DN().

If gRPC is serving a TLS conn, we can get the connection state with

if peer, ok := peer.FromContext(ctx); ok {
    if tlsInfo, ok := peer.AuthInfo.(credentials.TLSInfo); ok {
        // work with tlsInfo.State
    }
}

For gRPC-Gateway we can forward headers (set by Echo middleware) using runtime.WithMetadata in pkg/rpcserver

[johanstokking]

For the gRPC is serving a TLS conn, indeed that's the plan and accounted for that.

The rest through headers should work by composing the DN from the headers.

[johanstokking]

Let's start thinking about how we want to implement this.

I think that we should aim for having TLS termination by the proxy only. TTS middleware checks for headers set by the proxy. The reason is simplification of our stack and making better use of proxies, including smarter rate limits.

I removed all TLS stuff from Packet Broker Router for this reason too, and I'm happy with the result. It now checks the Envoy and Traefik headers.

The following requirements apply:

1. Dynamic set of trusted CAs. This can be ours and this can be a third party, like a gateway vendor or network operator, or a mix of this
2. Relate client certificate to entity rights
   • Directly from the subject (alternative names); i.e. the gateway EUI. The middleware would still have to look up the gateway identifiers by EUI (and check if the tenant matches)
   • Via a mapping of certificate fingerprint to entity rights
   • Via the trusted CA only, i.e. The Things Join Server accepts any client certificate signed by the CA of the NetID

[htdvisser]

Let's wait with this until we put Traefik in front (https://github.com/TheThingsIndustries/lorawan-stack/issues/2185)

[htdvisser]

Bumping this back to triage. Do we still want to do this? And how much priority does it have? I don't remember hearing about any concrete demand for this recently.

cc: @NicolasMrad

[johanstokking]

At some point we'll need this for connecting gateways using mutual TLS. The gateway to LNS protocol that the LoRa Alliance is working on is most likely using mutual TLS as its primary mechanism for authentication.

We would be trusting CAs from gateway vendors by default, but also custom CAs based on the TTS level (and tenant level). We might just combine the known gateway vendor CAs with the CA pool configuration that we already have. We just need tenant-level settings in TTSE for selecting which gateway vendor CAs are trusted and an additional CA pool for custom CAs.

I believe we would prefer the proxy to terminate TLS, even if that means that we need to support various proxy-specific header checks to verify if the certificate is actually valid.

I don't think we need mutual TLS for other services than connecting gateways.

[johanstokking]

Let's triage this again as it is getting relevant:

1. Azure Sphere will issue certificates to gateways, that they use to authenticate with external services (like LNS). The CA is per Azure customer account. This means that we need some contextual configuration of the CA pool
2. LoRa Alliance GW-NS interface is heading towards mTLS support as well, where gateway vendors could preprovision gateways with certs. In that case, we could trust gateway vendor CAs

[htdvisser]

If we're bumping this back to triage, let's also see who else can pick this up. I don't think I'll have time to look into this any time soon.

[KrishnaIyer]

🤚

[johanstokking]

@NicolasMrad what is this blocked on? Please write here what's been discussed.

References https://github.com/TheThingsIndustries/lorawan-stack-aws/issues/240

[KrishnaIyer]

This is being tracked in internal issues/PRs. Closing this one

[aniston]

This is being tracked in internal issues/PRs. Closing this one

from the outside ;) any update on this ?

it does interest me from the traefik point of view, as of TTS v3.22.2 with traefik v2.x on docker, a Gateway trying to join with Basics Station + LNS method (Auth+Token) still cannot join. If there is a solution via traefik , can you please share it.

[KrishnaIyer]

We have TLS support recently merged but not yet deployed. This feature will not be available for the open source deployments.

a Gateway trying to join with Basics Station + LNS method (Auth+Token) still cannot join

Authentication via an auth token does not depend on the external proxy settings. So your error is else where. Please ask on the forum.