Replace Dex with Canonical Identity Platform

[ca-scribner]

Context

This is a tracking issue for replacing Dex in the Charmed Kubeflow bundle with the Canonical Identity Platform. The goal of this effort is to no longer provide an authentication integration solution in Charmed Kubeflow itself, but instead to rely on a general solution provided by the Canonical Identity team.

What needs to get done

1. spec our current authentication flow, so we have a source of truth internally and a document to speak around when talking with the identity team
2. deploy the identity bundle to gain experience with it
3. do an MVP implementation to identify any challenges (general consensus is that a solution is feasible, but nobody has a complete understanding so we need to actually try to implement something)
4. design how the Kubeflow and Identity ingresses should be laid out (ex: should Kubeflow ingress provide a route to Identity? Should Identity have its own ingress?).
5. confirm that any existing customer solutions will be supportable with the Canonical Identity Platform (eg: does the Identity Platform support the external identity provider they use, and is there a viable migration path for upgrade)
6. spec the user experience for both enterprise deployments (deployments for multiple users, served on a company DNS, etc) and single-user deployments (deployed on a laptop, no public certs, etc)
7. design the migration path for existing single and enterprise users
8. design how Charmed Kubeflow integration tests should look if we use the Identity Platform solution

Definition of Done

1. Charmed Kubeflow uses the Canonical Identity Platform for integrating with authentication providers

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5228.

This message was autogenerated

[ca-scribner]

Notes from an initial meeting between Kubeflow and Identity teams:

Details of replacing Dex with Identity Platform

• Identity team thinks we should be able to plug the Identity Platform in where Dex is currently used.
  • configuring oidc-authservice and Identity Platform to work together and integrate with Istio sounds relatively easy
  • main challenges:
  • handling ingresses and TLS
  • Note: This sounded like one of the hardest parts of the integration. There will be some heavy lifting to do here
  • Identity Platform uses Traefik as an ingress and expects https communication throughout. Both Identity and Kubeflow will need to be accessible through an ingress, so we need to design that layout (should both have their own ingress? should one sit under the other? how will the certs be handled?).
  • there is probably some passing of certificates that needs to happen behind the scenes to ensure https communication works
  • There may be some inspiration available in the grafana integration
  • example of how grafana+identity integrate
• what is the deployment experience like? do we deploy Identity Platform in a separate model, similar to observability?

Integration Testing

• the Identity Platform is not lightweight, and likely not suitable for use in Kubeflow charm integration tests. We'd need something other than Identity Platform for any integration test that needs auth (which, atm, is any test that needs to test through an ingress, like our dashboards or notebooks)
  • a possible solution could be developing a true single-user experience for Charmed Kubeflow. So for single-user deployments, something that removes the need for authentication at all (somehow it just sets the kubeflow-userid header to a fixed value for all traffic and users don't need to worry about it). If we had this, we could use the single-user experience for most charm integration tests, and only deploy the full Identity bundle in select cases.
  • Alessandro mentioned with Grafana integration they had to do something lightweight. We might be able to follow that (I think he said they use dex, which might just be what we're already doing...)

Migration/Upgrade

• we will need a migration path for:
  • single-user deployments (someone deployed on their laptop and created a single user in dex)
  • enterprise deployments (someone who connects to an identity provider)
• need to handle the actual handover (likely our oidc-authservice cannot support two of these at once, so we'd need to have a way to switch. Maybe by deploying another ingress?)

Replacing oidc-authservice with something from Identity team

• a long-term goal, but not feasible by 2024-04 and may not be feasible by 2024-10 (Kubeflow team would need to advocate for it)
• likely requires a rewrite of the actual backend providing this function

Logistics/Planning

• Identity team:
  • has capacity in the current and upcoming sprints to support the Kubeflow with replacing dex with the Identity Platform. The expectation there is that some involvement is needed from the Identity team, but not a huge effort
  • does not have capacity in the current sprint to support replacing the oidc-authservice workload with a solution from the Identity team. There may be capacity for next sprint, but we need to advocate for that

Actions

• [ ] @kimwnasptd: If we want to replace oidc-gatekeeper next sprint, ask for that in Identity's roadmap during initial planning

[kimwnasptd]

Thanks for the summary!

IMO the main pain point we have right now is that the AuthService is unmaintained. So unless we would have a solution that would replace AuthService as well, I don't see much value in just replacing Dex. There's not burning need or issues with Dex right now from our side.

Then some concerns I had from the above:

1. https from browser: This will create a very bad UX for users that just spin out KF to play around locally
2. in-cluster tls: This cycle we are going full on with more Istio support, which also includes mTLS between the pods in the Istio mesh. Then there's going to be the mesh team that will also take a stab at the mTLS problem. I feel it's too soon to commit, since the moving pieces we need (Istio down the road from another team) might clash with decisions currently in the Identity Charms
3. Ingress story: Another pain point, as we'd either
   1. have to put work on Istio IngressGateway charm to be closer to what the rest ingress charms are doing, so that we can switch Traefik with it
   2. Put Traefik in front, and remove authservice/dex, and all authnz logic is plugged in Traefik. Which then redirects the logged in traffic to Istio IngressGateway

So from our side, in my mind, the requirements to integrate with Identity team's charms are:

1. Have the option to have http to the Traefik Ingress, when used with Identity team's charms
   • And be able to have static users
2. Fully replace our current Istio->AuthService/Dex flow Traefik and Identity team's charms
3. Expose Traefik in-front of Istio IngressGateway, and be able to send all traffic to Istio (IMO it's not worth it for us to invest on changing the Istio IngressGateway charm)

cc @ca-scribner @DnPlas @shipperizer

[shipperizer]

@kimwnasptd

1. I think we can kinda fix that by either having a good local ACME setup for certificates or we could add some machinery to accept self signed certificates and create a local DNS record in the /etc/hosts file other solution would be to have it all in dev mode, IDP included (as pointed out by @nsklikas and Massi) but this makes me wonder if that is the use case we want to cater for I do think that for some stuff our solution is overkill (local development) but for other it should be the go-to one (production)

2. Using a publicly trusted certificate provider would bypass the problem but as said, this is true in production deployments I guess, locally it won't unless someone is happy to depart with 10$ a year minimum for their own domain

3. Networking wise, I don't see a major problem in getting it working as long as the requirements we have for istio deployment (ability to spin up a load balancer, DNS management and domain ownership) can be extended to traefik

anyway, happy to have another chat if needed