Extend Teleport to OAuth2 apps (Grafana, Happa, Prometheus, etc.)

[gawertm]

### Tasks
- [ ] https://github.com/giantswarm/roadmap/issues/3219
- [ ] https://github.com/giantswarm/roadmap/issues/3220

### Blockers
- [ ] Teleport integration with Happa API for Kubernetes API access. See more details [here](https://gigantic.slack.com/archives/C053JHJC99Q/p1708517563819189) and [here](https://gigantic.slack.com/archives/C02GDJJ68Q1/p1708508263922009?thread_ts=1706692097.776539&cid=C02GDJJ68Q1)
- [ ] Grafana Loki DS
- [ ] Dex integration

[tuladhar]

Discusssion topics:

• Teleport application access conflicts with Dex.
• Grafana is used by customer, which implies customer will have to use Teleport to access the Grafana.
• Happa - Teleport does not handle user session, similar to how Dex does, for e.g: who logged in as.

[tuladhar]

Prometheus UI access via Teleport:

• https://prometheus-golem.teleport.giantswarm.io

Access flow (without nginx ingress/dex in the mix):]

User -> Teleport Cluster (login)
         -> Teleport App Agent (running on golem)
             -> Prometheus internal svc endpoint

Route53 entry (manual for now, until we have wildcard entry *.teleport.giantswarm.io):

prometheus-golem.teleport.giantswarm.io -- points --> Teleport Cluster Proxy ALB

Here's the teleport app config:

> cat prometheus-golem-app.yaml
kind: app
version: v3
metadata:
  name: prometheus-golem
  description: "Prometheus - golem"
  labels:
    env: test
spec:
  uri: http://prometheus-operated.golem-prometheus.svc.cluster.local:9090

[tuladhar]

Document explaining Prometheus access setup:

• https://docs.google.com/document/d/1t3YnhUmu0UOSNhw5kr8GBLRJjuZcDf_kGXXlaAP7dDE/edit#heading=h.8vev9pjz9n7p

[tuladhar]

[tuladhar]

Plan B worked - grafana-golem.teleport.giantswarm.io

Plan A didn’t work, and is not backward compatible with customer, as there are breaking changes when setting up dex behind Teleport.

Why Plan A didn't work?

• Starting off, we have to update GF_AUTH_GENERIC_OAUTH_AUTH_URL=http://dex-golem.teleport.giantswarm.io/ for Grafana, otherwise, it will keep redirecting us to VPN accessible endpoint dex.golem.gaws.gigantic.io
• After that, dex login works using Teleport-powered Dex endpoint, but in Grafana, we get Login failed: Missing saved oauth state.

• And to fix that, we need to update Grafana GF_SERVER_ROOT_URL env to https://grafana-golem.teleport.giantswarm.io/

• But, after changing GF_SERVER_ROOT_URL we hit this issue:

• To fix that, we need to update dex-operator secret to add https://grafana-golem.teleport.giantswarm.io/ to list of redirectURIs:

  redirectURIs:
  - https://grafana.golem.gaws.gigantic.io/login/generic_oauth
  - https://grafana-golem.teleport.giantswarm.io/login/generic_oauth
  name: grafana

• Then, we get Grafana with Dex login behind Teleport. But, with all those changes, dex login for customer breaks.

How Plan B worked?

• With B, no changes needed on dex side, and normal customer dex login flow works the same. The only change happens at Grafana side by adding this to configmap:

  [auth.jwt]
  enabled = true
  auto_sign_up = true
  header_name = Teleport-Jwt-Assertion
  username_claim = sub
  jwk_set_url = https://teleport.giantswarm.io/.well-known/jwks.json

• All request from Teleport, includes JWT token pass-through and sent directly to Grafana pod.

• Grafana access via existing ingress grafana.golem.gaws.gigantic.io works the same.

[anvddriesch]

Since customers dex access can not be affected by either scenario, we are unable to change settings like the OIDC issuer flags on the kubernetes API server or the oauth provider flags for grafana. (both allow only one issuer) This is not an issue in scenario B since we can use the jwt token directly and bypass dex completely. However, it is an issue in scenario A for the reasons described in Purus comment. The limitation on issuers is the reason we were using dex in the first place. We may have a bit more freedom with 1.29 structured auth config options but we have not tested this yet. As of right now, running dex inside teleport to make it available to us does not seem possible if we don't also want to make dex unavailable to customers. We already have a limitation on reproducing dex issues due to identity provider configuration on the customers side. Bypassing dex and removing the VPN will grow this limitation since dex will become unavailable to us. However, since we already need to debug in tandem with customers it may not be a huge change.

[tuladhar]

Honeybadger Feedback (Happa)

• Feedback #1 - How does local setup works with Teleport?

  • Resolved: By running local teleport app agent pointing to local Happa instance, confirmed with Dmitry, it worked.

• Feedback #2 - Is there a way to get a token not from a header? Separate api endpoint maybe? the problem with the header is that we cannot access it from client side (from JS). We would need to have a server to transfer it to the client, but we don't have a server for happa.

  • Teleport automatically injects JWT tokens into the headers of requests going through the Application Access proxy, it doesn't seem to provide an endpoint for directly accessing the JWT token. There is only JWKS endpoint for verifying the token - https://teleport.giantswarm.io/.well-known/jwks.json.
    • Asked in Teleport community Slack channel.

[tuladhar]

Atlas Feedback (Grafana/Loki)

• Feedback #1 - If I use grafana through teleport, the loki datasource which forward the OAuth token is failing with the current loki-multi-tenant-proxy implementation. On the other hand, if I use grafana through https://grafana.golem.gaws.gigantic.io/ it works fine. Which OAuth token are you provided to grafana ? azure AD?
  • Grafana access via Teleport, doesn't send oAuth token instead JWT token is sent, as we don't go through Dex.
  • I have setup a call with Atlas to discuss more on this.

[tuladhar]

Atlas - Challenge #1: Loki DS uses oAuth token forward

• Problem: Since, login to Grafana uses Teleport JWT token, oAuth token setup for Loki DS breaks.
• Potential solution: Forward Teleport-Jwt-Assertion header from Loki DS. Atlas is looking into it.

[tuladhar]

Honeybadger - Challenge #1 - Happa can't accept Teleport Jwt token

• Problem Since, Happa is fully client side without server component, Dmitry confirmed that it can't access Teleport JWT token from the header passed on by Teleport. And, teleport doesn't offer an endpoint to access the JWT token directly.

• Potential solution: Re-configure nginx with an endpoint /jwt and then utilize Lua module to read the Teleport-Jwt-Assertion header and return as response.

  load_module modules/ndk_http_module.so;
  load_module modules/ngx_http_lua_module.so;
  
  location /jwt {
  add_header Content-type text/plain;
  
  content_by_lua_block {
    ngx.say(ngx.req.get_headers()['Teleport-Jwt-Assertion'])
  }
  }

Honeybadger - Challenge #2 - Athena /graphql endpoint

• Problem Happa UI sends request to athena endpoint and putting athena endpoint behind Teleport breaks the request.

• Potential solution: Expose athena /graphql endpoint through happa ingress and configure happa to use happa/graphql to reach athena.

[tuladhar]

Discussion with Honeybadger

Slides: https://docs.google.com/presentation/d/1OLtCijqp6VM1Xu2p0sxvyr1d2NJD6wHK0ypQ1mzRRCA/edit?usp=sharing

Action Items:

• BigMac to delivery https://github.com/giantswarm/roadmap/issues/3220 and Honeybadger can then start experimenting and make changes to Happa.
  • Dmitry, will look into webpack webserver to expose the JWT token endpoint for local development. We will assist if help is needed.
  • BigMac to setup happaapi behind Teleport, and Honeybadger will start experiementing and make changes to Happa API.
• DevPortal is low priority, and for now, we won't move it behind Teleport until we're done with Happa.
• BigMac to update opsctl to adjust opsctl open, where for CAPI it should go through teleport powered happa endpoint (e.g: happa-golem.teleport.giantswarm.io), and for others, dex powered happa.

[tuladhar]

Discussion with Dmitry

Attendees: Puru, Spyros, Antonia, Pawel, Dmitry

Action Items:

• BigMac blocked on Happa teleport integration due to Happa K8s API access
• BigMac blocked on Backstage teleport integration due to Dex
• Dmitry to discuss with honeybadger about “Happa backend” and research on “OIDC” vs “service account” for Backstage K8s API access, Backstage permission framework.
• BigMac to help with the review.

[tuladhar]

SOCKS5 proxy as an alternative

@gawertm It seems we can leverage the SOCKS5 proxy feature of Teleport SSH to access internal web apps. I have documented the setup process in intranet. I posted the demo in our channel.

[gawertm]

@tuladhar thanks, I just saw that. Can you draw some network diagram how the traffic flows with the socks5 proxy? I want to fully understand :) and also about this:

Only use proxy for accessing internal web apps, because potentially, we can use it to browse other external websites.

[tuladhar]

@gawertm Here's the network diagram. Let me know if this doesn't make sense :)

Only use proxy for accessing internal web apps, because potentially, we can use it to browse other external websites.

Yes, think of SOCKS5 proxy as VPN through SSH connection. So, any network access the node where SOCKS5 is established on, we can access them too, which means access to the internet if node has access to it along with internal network access.

[gawertm]

ok understood, thanks! lets discuss it in the refinement today :)

[gawertm]

we came to the conclusion, that this solution is not user friendly enough and too much config is needed upfront. While it is a good emergency access solution, we will still keep the VPN for App access in private installation. to be confirmed with @alex-dabija. Also we will revisit with kubernetes 1.29 and structured auth config

[alex-dabija]

We agreed in SIG Architecture to keep the the VPN for now.