Spike: support `client-go` `auth-provider`s

[gcapizzi]

Background

We now believe that the best way to support authentication against all different types of Kubernetes clusters is to use $KUBECONFIG. The explore describes the different types of authentication $KUBECONFIG supports. We believe we can support all of them (except the deprecated ones).

Questions

Among the different auth-provider configs, OIDC is the only one which is not getting deprecated. We believe we can support it, as it stores the ID and refresh tokens we're used to, but we want to make sure we can do this fully. In particular, we want to confirm that we can refresh the token when necessary. Obtaining the token is out of scope, as it is assumed that the user has already done it at this point.

[gcapizzi]

Instead of trying to support the oidc auth-provider in particular, today we managed to find a way to support all auth-providers generically!

auth-providers implement an AuthProvider interface which has a WrapTransport method, taking an http.RoundTripper and wrapping it. We found this to be remarkably similar to the ConnectionWrapper interface in the CF CLI! The ConnectionWrapper wraps a Connection, which has a very similar interface to http.RoundTripper, so we were able to implement ConnectionRoundTripper, an adapter from Connection to RoundTripper. This could then be wrapped with AuthProvider.WrapTransport() and then used!

This proved to work with the gcp auth-provider and we believe it would work with all of them, including oidc. This approach allows to reuse a lot of code, including persisting tokens in $KUBECONFIG, which needs to be protected by locks to avoid races.

At the end of the day, we looked at using a similar approach for exec plugins, but we had no luck. The exec plugins modify the TCP configuration of the HTTP client to be able to inject client certificates, and getting those out seems complicated. We might give this a second shot but our previous approach, invoking the executable directly, looks simpler. On the other hand, the client-go implementation comes with a lot of interesting functionality (e.g. caching) that we don't have (yet).

[kieron-dev]

Update on re-using the client-go exec authenticator code

It is possible to use the code from exec.go. The interface is different from the auth-provider code we borrowed above. In this case, we must invoke UpdateTransportConfig passing a *transport.Config that we control.

This does two things. First it sets the TLS.GetCert property on the transport config to grab the certificate, if any, from the credentials plugin. Secondly it sets up a wrapper function so a RoundTripper can be wrapped by one providing the bearer token from the credentials plugin.

Caching and execution of the credentials plugin is taken care of, so we don't have to worry about that. If the user has to login from the GetCert() trigger, a second login will not be required when getting the bearer token.

We can use the *transport.Config object to retrieve the client certificate, if any, by calling GetCert(). Unfortunately, the PEM cert and key have already been parsed and converted into an x509.Certificate. We need them back in PEM format to configure the k8s client, so there is the messy process of reversing that transformation.

We can also invoke the WrapTransport method on the *transport.Config object to use the ConnectionRoundTripper built for the auth-provider work. This will add the bearer token if the credentials plugin returned one, and if the authorization header is not already set.

So overall, this appears quite reasonable. The code is slightly verbose given the string of potential errors returned, but using *transport.Config doesn't feel that hacky!

[kieron-dev]

Note that we can test the client cert part of the credentials plugin by using a pinniped configuration. And to test a credentials plugin that produces a bearer token, targeting a GKE cluster and switching the config to use https://github.com/sl1pm4t/gcp-exec-creds works.