Add support for workload cluster OIDC auth via `kubectl gs`

[snizhana-dynnyk]

Towards Migration to Management API

User stories

• As a user of kubectl, I want to quickly set up a context in my kubectl config file to access a workload cluster that supports OIDC authentication.

Description

The kubectl gs login command already supports OIDC authentication for the Kubernetes API of a management cluster (a.k.a Management API). This story here is about allowing roughly the same for a workload cluster.

Questions

• Will this require dex to be deployed to the workload cluster? YES
• With client certificates, a common use case is for an admin to create a self-contained kubectl config file for another person or for use in an automation. Is this / should this also be possible with OIDC? How? NO

---

TODO

• [x] Refinement
• [x] #1063
• [x] #1064
• [x] #1078
• [x] #1065
• [x] #1097
• [x] #1098
• [ ] #1367
• [x] #1156
• [ ] #1368
• [ ] Look into replacing k8s-authenticator with athena and kgs

[marians]

Added some details to the main comment

[marians]

One problem to solve: the client has to trust the workload cluster CA certificate. For the management cluster we have athena delivering the CA certificate, so that it can then be added to the server entry in the kubectl config.

[TobiasLierzer]

I've requested this as a convenience feature for the everyday work. So with regards to the second question, I would not see the requirement to create automation configs / configs for another person through this process.

[ghost]

Assumption:

1. WC is already prepared and Dex and ingress controller is running

As an Admin with MC access, I want to quickly login to a WC.

1. That is to make kubectl gs login work with WC with a minimal set of OIDC flags
2. WC trusts the certificate from the MC (using the fetch client certificate mechanism) - we create secret from the cert-config.

As an Admin, I want to provide access to a WC

1. The command is handed to a user so as to use the OIDC provider to login into the WC
2. In here, the dex host (domain name) could be configurable to over-ride the default by using flags

Additional feature Get the command to use, (e.g cli guide) from happa (can be in an access tab or cluster section)

[pipo02mix]

We have another customer in Openstack implementation that wants to use kubectl gs to make easier the access to the cluster

[anvddriesch]

Current state: Install dex-app in WC and use k8s-authenticator which allows OIDC by copying generated commands. Cumbersome.

Goal: Support logging into WC using dex via kubectl-gs login. Support existing flags of kubectl gs login

Non-goals Making dex default on all WCs Creating self contained kubeconfigs to hand out to third parties (SSO should be used for a specific user)

Notes from refinement We have this for MCs, where we run athena which exposes the CA. Where should this come from in the WC?

• We need a way to inject this independently for each WC
• We should not have to need MC access for this
• Can we use athena? Could this have security implications?
• Alternatively to athena we could use a simple web service
• How does it work for the k8s-authenticator? Do we need to add the CA manually in the values.yaml
• How do we get the CA to present via a service? Is it sealed via vault in legacy clusters?
• Does it make sense to support this for legacy? Either way, focus should be CAPI
• Is this something we want to offer as a default? What is the workflow here?
• What is needed to do it by default? Customers would need to supply the callback URLs etc for each new cluster

[pipo02mix]

My 2 cents

Does it make sense to support this for legacy? Either way, focus should be CAPI

I would not invest time in legacy, by now only customer wants it is in CAPI

[marians]

Open questions / tasks from refinement

• Provide Athena as a managed app for workload clusters
  • Must be available in a shared catalog (we want it to be in giantswarm)
  • Values from giantswarm/config cannot be relied on
  • How does the user installing/configuring the app provide the CA certificate?
  • For CAPI: a secret containing the CA cert is stored on the management cluster. We can provide a kubectl command to retrieve this. Also we could display this in happa, in case the user has the according permissions.
  • For non-CAPI it may require support to help out.
• dex-app config distinguishes between workload and management cluster
  • OIDC client-id currently used by kubectl-gs and happa is only permitted if management cluster. We must also allows kubectl-gs's client-id in the case of a workload cluster.
• Would it make sense to provide dex-app and athena (and maybe even cert-manager, nginx-ingress-controller) as an app bundle? Depends on how app bundles will work e. g. if one of the contained apps is already installed individually. -> bring this to honeybadger
• kubectl gs login needs a UX review when enabling OIDC for workload clusters. The --workload-cluster flag will be confusing.

[pipo02mix]

How does the user installing/configuring the app provide the CA certificate?

In legacy cluster operator set the CA in cluster values configmap, in CAPI cluster apps operator does it.

[anvddriesch]

Created issues from the remaining points but will close this since it's done.