Skipper mTLS support

[abinet]

Hi all,

does skipper support mTLS use-case?

Nginx ingress controller use following annotations to configure client validation:

nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
nginx.ingress.kubernetes.io/auth-tls-secret: "default/mycerts"

Is it possible to get similar with skipper?

[szuecs]

@abinet not as flexible as nginx annotations, no. We have only flags like:

  -client-tls-cert string
        TLS certificate files for backend connections, multiple keys may be given comma separated - the order must match the keys
  -client-tls-key string
        TLS Key file for backend connections, multiple keys may be given comma separated - the order must match the certs
        ignore the verification of TLS certificates for etcd
        flag indicating to ignore the verification of the TLS certificates of the backend services

This is not useful for the ingress case.

Right now the most useful deployment is to use something like Amazon ALB/NLB in front and terminate TLS there.

[aryszka]

It could be a nice feature for Skipper to support mTLS, and it doesn't seem too hard to implement, either. See the ClientAuth field of the crypto/tls.Config

It may be more tricky to accept different client certificates for different routes, but we can also implement that in a separate phase. It seems possible, because the TLS field of the net/http.Request exposes the crypto/tls.ConnectionState, which exposes the crypto/tls.Certificate via the PeerCertificates field, and this way we can use predicates to only select a given route for a desired client.

But I cannot tell when would implement this. PRs are of course welcome.

[szuecs]

@aryszka the other problem to solve is that we have to update also TLS certificates in the server handler, because with mTLS you need to terminate TLS in the server process.

[szuecs]

@AlexanderYastrebov I don't think it's mTLS authnz is implemented only server handler TLS.

[containerpope]

@szuecs we are currently looking into replacing traefik with skipper. One use case we have is to extract additional client certificates from the http request. In traefik we use the pass_tls_client middleware to forward the client cert in a header to our backend services. We do not want to validate them in skipper, just forward them as header.

In traefik I can achieve this with the following middleware: https://doc.traefik.io/traefik/middlewares/http/passtlsclientcert/#configuration-examples

Is there a way to use an existing filter which can access this data from the request and forward it as header? Alternatively, could I develop a custom go or lua script based filter for this?

The following code is used to access the cert in traefik:

func (p *passTLSClientCert) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
    logger := middlewares.GetLogger(req.Context(), p.name, typeName)
    ctx := logger.WithContext(req.Context())

    if p.pem {
        if req.TLS != nil && len(req.TLS.PeerCertificates) > 0 {
            req.Header.Set(xForwardedTLSClientCert, getCertificates(ctx, req.TLS.PeerCertificates))
        } else {
            logger.Warn().Msg("Tried to extract a certificate on a request without mutual TLS")
        }
    }

https://github.com/traefik/traefik/blob/master/pkg/middlewares/passtlsclientcert/pass_tls_client_cert.go

[szuecs]

@containerpope interesting problem. Right now we don't have a filter that would do this, but it seems simple to implement. It seems that you basically have to read request data and filters have:

• Request(FilterContext)
• Response(FilterContext)

FilterContext has access to the request like ctx.Request(), which is the request we got from Go stdlib via the http/https handler. see also https://opensource.zalando.com/skipper/tutorials/development/#filters

A filter use in eskip would be something like: tlsDataPassToHeader(x,y,"X-My-Header") Name TBD and args x,y are whatever you need to pick the right data you want to get from the request.TLS.

Would you like to create a PR to add this feature?

[szuecs]

Others do

ingess-nginx

Looking at https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#client-certificate-authentication , I think we do not want to have an option to not validate the cert, because it makes no sense to not validate. There is an idea about validating CNs and DNs as authz. I think CN makes most sense, but if someone wants to have DN it would not be a problem to add.

envoy

https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/client_ssl_auth_filter#config-network-filters-client-ssl-auth and https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl#arch-overview-ssl-auth-filter As always the envoy configuration reads a bit strange to me. I don't see a relevant feature that they provide. They have links to SPIFFE which is mostly a project to do cert rotation

ghostunnel

https://github.com/ghostunnel/ghostunnel is an mTLS proxy written in Go. It support cert rotation, SPIFFE, ... and has Apache license. I guess this one will be the most interesting to read, because it supports client and server side of the mTLS.
They support ACME and cert hot swapping. I think this will be an inspiration for the future if we want to implement more features like that.

SPIFFE

https://spiffe.io/docs/latest/spiffe-about/overview/ https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/

They have a SPIFFE ID, for example spiffe://my-org.example/my-application. which is part of an SVID. SVID can be either x509 cert or a JWT.

An example how this would work together with envoy and SPIRE you can read https://blog.envoyproxy.io/securing-the-service-mesh-with-spire-0-3-abb45cd79810. In the end it would issue a x509 cert as SVID with a SPIFFE ID in the SAN field. This suggests we should be able to validate SAN as authz filter.

Another example is pinterst blogpost, that shows an example of client x509 cert with SAN:

X509v3 Subject Alternative Name:
  URI:spiffe://pinterest.com/service_a,
  DNS:service_a.pinterest.com

So SAN field needs to be prefixed with URI: for example. Maybe we should have a prefix for the filter arguments. They also have a secret-manager service called knox which is written in Go.

SPIRE has a go library integration to talk to SPIRE server running somewhere.

Initial implementation

Thinking about authentication, we need to check that the client cert is a valid cert issued by a CA that is public trusted or our trusted CA that is passed to skipper as option. I think it's fine to have a flag/config for a single CA to validate client certs. The CA will be passed to the filter spec creation and a copy will be stored in each filter instance. If we would one day need to be able to rotate it we can add a new spec creation that use secrets module to read from file.

For authorization, I think we want to validate client CN, which would need to be possibly different for each route. We could achieve this as list of CNs which are allowed to connect to us for example. Filter input will be either varargs (not possible to add more options in the future) or a string with some separator like ,, for example "cn1,cn2,cn3".

We could either split both mentioned functionality into 2 filters or have one. The first one should do the authn part and the second the authz part. The authn filter could store some data in the stateBag and pass it like this to the authz filter, that would just read data from stateBag.

I would propose to split it into 2 filters:

• tlsAuthn() which gets the CA and validates the cert and writes everything we need for authz filters from the parsed client cert into a stateBag. We won't implement OCSP nor revocation. Revocation can be done by passing a new CA as flag or wait until cert is not valid by time anymore. If there is a need we can always think about how to have a background refresher goroutine to fetch revocations, but we won't fetch a revocation list in the hot path.
• tlsAuthzCN("cn1","cn2",...) I would propose to have varargs, because the filter will only check for valid CNs that are allowed to pass the route. If we want to check something else we would add a new filter to check the new data, so I don't expect that we need more data to add as options for the filter.
• tlsAuthzSAN("URL:", "e1","e2", ..) I would propose a prefix to strip and varargs that can be used to validate a list of allowed SAN entries to connect to the route. Similar to the CN filter I would not think it will be useful to add more options to the filter.

[containerpope]

@szuecs really fast and cool implementation, thanks a lot! The pass cert as header is actually used to validate the certificate behind skipper. It’s just a simple forward that data to my service. Unpacking the whole client cert validation stuff might be something related to this issue: https://github.com/zalando/skipper/issues/1780

[mjungsbluth]

@szuecs the proposed filters make a lot of sense and thanks for the great writeup!

What I have seen in the past is that as CAs have become easier to manage and setup, each trust domain/boundary gets its own (intermediary) CA which is relatively short lived (weeks) and that those intermediary CAs are actually used as trust anchors instead of a root CA. If you go for a zero trust model you might not even have a root CA but just local CAs that are living per trust domain (for example one Kubernetes Cluster).

Implementation wise this would mean that multiple CAs would need to be configured and also that reloading of a new certificate would need to be supported. Both are things that can be done later but maybe the implementation can already foresee that these cases exist.

In terms of filters this might require another filter (sth like tlsAuthZIssuerDN) to make sure only a specific CA is actually expected for a specific route.

When looking at how SPIFFE is often used, you might want to put a predicate on the trust domain and not care about the workload in an URI SAN. Maybe it would be better to change the filter to `tlsAuthzSANURI("my.trust.domain", "my/path/*") and allow wildcards in both the host and path part of the uri (or have the SAN type as the first parameter) . This would also require equivalent filters for other types of SANs (DNS, Email, IP) if they are to be supported or explicit support for those types.

[pioneerit]

Talking about mTLS, I just learn that end of March 2024, AWS added mTLS support to ALB. https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-mtls-for-application-load-balancer/

Even if it's not directly Ingress solution, and we're prefer the NLB when possible, it's still some kind of "Others do".

Update: the mTLS released before Re:Invent in November 2023, aws news post