Migrating from jetstack to tremolosecurity failed

phr0zenx commented 2 years ago

I'm using the helm chart from tremolo's kube-oidc-proxy branch: https://github.com/TremoloSecurity/kube-oidc-proxy/tree/master/deploy/charts/kube-oidc-proxy Need your advice on what i'm missing?

Forwarded from following url: https://www.reddit.com/r/kubernetes/comments/rvxqjt/updating_kubeoidcproxy/

Encountered following error:

I0113 11:30:21.271756 7640 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 24 ms Duration 32 ms
I0113 11:30:21.271756 7640 round_trippers.go:577] Response Headers:
I0113 11:30:21.272732 7640 round_trippers.go:580] Date: Thu, 13 Jan 2022 03:30:21 GMT
I0113 11:30:21.273728 7640 round_trippers.go:580] Content-Type: text/html
I0113 11:30:21.274723 7640 round_trippers.go:580] Content-Length: 146
I0113 11:30:21.275720 7640 round_trippers.go:580] Strict-Transport-Security: max-age=15724800; includeSubDomains
I0113 11:30:21.285723 7640 request.go:1181] Response Body: <html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
I0113 11:30:21.288718 7640 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0113 11:30:21.291721 7640 cached_discovery.go:125] skipped caching discovery info, no groups found

Using keycloak as our oidc. Value.yml file for helm deployment as follows (same file can be used to successfully deploy jetstack's helm chart):

replicaCount: 1

image:
  repository: quay.io/jetstack/kube-oidc-proxy
  tag: v0.3.0
  pullPolicy: IfNotPresent

oidc:
  clientId: REDACTED
  issuerUrl: https://REDACTED/auth/realms/master
  usernameClaim: email
  usernamePrefix: "oidc:"
  groupsClaim: groups
  groupsPrefix: "oidc:"

  signingAlgs:
    - RS256
  requiredClaims: {}

tokenPassthrough:
  enabled: false
  audiences: []

extraImpersonationHeaders:
  clientIP: true

extraArgs: {}

extraVolumeMounts: {}

extraVolumes: {}

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
  hosts:
    - host: REDACTED
      paths:
        - /
  tls:
    - secretName: REDACTED
      hosts:
        - REDACTED

podDisruptionBudget:
  enabled: false
  minAvailable: 1

resources: {}

initContainers: []

nodeSelector: {}

tolerations: []

affinity: {}

mlbiam commented 2 years ago

Sorry, missed a notification. We didn't spend much time on the helm charts, since our focus is on our integration with openunison's charts. That said, it looks like you are still using the old image

repository: quay.io/jetstack/kube-oidc-proxy

The new image is docker.io/tremolosecurity/kube-oidc-proxy:latest

It looks like the proxy is getting a 404 trying to start. Is the container in a read state?

kingdonb commented 2 years ago

Will you publish semver tags starting with the next release? Many clusters have a policy against deploying anything with a :latest tag. (Thanks for the hint!)

kingdonb commented 2 years ago

I tried to drop in the latest image on my cluster, which was also running the old jetstack kube-oidc-proxy from a Helm chart (that has been upgraded slightly) and got these errors preventing startup:

Error: Failed to read the CA contents: data does not contain any valid RSA or ECDSA certificates

[... the help text]

kingdonb commented 2 years ago

I got the impression that OpenUnison and kube-oidc-proxy are meant to be used together. I might be missing something because I see the blog post suggests setting these values:

impersonation:
  use_jetstack: true
  jetstack_oidc_proxy_image: docker.io/tremolosecurity/kube-oidc-proxy:latest
  explicit_certificate_trust: true

but since I'm not using OpenUnison, these do not appear to be values that I can set.

Is there an entrypoint for using the updated kube-oidc-proxy with another OIDC provider? I've been using Keycloak and kube-oidc-provider together. I found a section in the docs that mentions keycloak, but I am not sure what OpenUnison provides at large and I get the sense that I have to read more docs and take in the full product to understand better.

I was really hoping for a way to drop-in an updated kube-oidc-proxy and keep on moving, I'll read more docs since there's bound to be something important I've missed by skipping over so much. I don't want to minimize any of the work you've done by suggesting "you should do what I need instead" but if this is a common use case, it would be really helpful to have a document that is oriented towards migrating to OpenUnison, from Keycloak, or with an existing Keycloak installation.

I'll be very happy to be pointed at any existing docs that might help me get moving in the right direction!

mlbiam commented 2 years ago

Will you publish semver tags starting with the next release? Many clusters have a policy against deploying anything with a :latest tag. (Thanks for the hint!)

We're not planning to right now. Since we moved kube-oidc-proxy to ubuntu, we've added kube-oidc-proxy to our container monitoring system so it's rebuilt whenever Canonical releases fixes for CVEs. We're not going to tag each of these releases separately or maintain older versions as new versions get released.

We do tag the source for each release though and attach it to this github repo. If you want to have specific tags, what we suggest is to import the kube-oidc-proxy into a private registry and tag it there.

mlbiam commented 2 years ago

I tried to drop in the latest image on my cluster, which was also running the old jetstack kube-oidc-proxy from a Helm chart (that has been upgraded slightly) and got these errors preventing startup: Error: Failed to read the CA contents: data does not contain any valid RSA or ECDSA certificates

This is a side effect of moving from alpine to ubuntu. In the next release we'll make sure that by default it's reading CA certs from the right location, but until then you can add --oidc-ca-file=/etc/ssl/certs/ca-certificates.crt to the args section of your Deployment's container

mlbiam commented 2 years ago

I got the impression that OpenUnison and kube-oidc-proxy are meant to be used together.

We've forked kube-oidc-proxy so we could provide better support for our customers using it, but there is nothing in kube-oidc-proxy that is specific to OpenUnison, nor will we add anything that will force you to use OpenUnison. We do our own testing with OpenUnison, but there's no reason other OpenID Connect identity providers, like KeyCloak, won't continue to work. We're not testing with any other identity providers though, so any improvements to the helm charts, teraform temlates, etc, will need to be from the community.

but since I'm not using OpenUnison, these do not appear to be values that I can set.

Correct, these are for OpenUnison. We then translate these into a Deployment for kube-oidc-proxy in our own helm charts. Everything is integrated in Helm.

Is there an entrypoint for using the updated kube-oidc-proxy with another OIDC provider?

No, kube-oidc-proxy doesn't know the difference between OpenUnison, KeyCloak, Dex, etc. It uses the OpenID Connect standards to discover proper endpoints, public keys, etc.

I was really hoping for a way to drop-in an updated kube-oidc-proxy and keep on moving

Minus the issue with the CA certs, its should be drop in. You'll need to make some RBAC updates to get impersonation working but other then that it's the same code, only with some new features.

it would be really helpful to have a document that is oriented towards migrating to OpenUnison, from Keycloak, or with an existing Keycloak installation.

While you don't need to migrate off of KeyCloak to work with Tremolo Security's fork of kube-oidc-proxy, I'd welcome you to take a look at OpenUnison. The biggest advantages of OpenUnison over KeyCloak are:

Pre-integrated kubectl command generator for quickly logging into k8s clusters from Nix and Windows, including CA certificates when needed
Zero configuration kubectl plugin for authentication
Pre-integration for securing the Kubernetes Dashboard
There is no database to manage or management UI to secure. Everything is K8s native through CRDs.

If you're interested, take a look at https://openunison.github.io/ for details about OpenUnison, also a comparison between OpenUnison, Dex, and KeyCloak - https://www.tremolosecurity.com/technology/kubernetes

We haven't produced a "migrating from keycloak to OpenUnison" guide, but if that's something you're interested in we can certainly take a look.

kingdonb commented 2 years ago

Thank you! --oidc-ca-file=/etc/ssl/certs/ca-certificates.crt

That solved it for me, I added in values:

   values:
+    image:
+      repository: docker.io/tremolosecurity/kube-oidc-proxy
+      tag: latest
+      pullPolicy: Always
+    extraArgs:
+      oidc-ca-file: /etc/ssl/certs/ca-certificates.crt

Using the latest tag will require people to use ImagePullPolicy: Always in order to be sure they are really pulling the latest version of the image – this is not advisable as you will quickly encounter Docker Hub rate limits, perhaps even as a paying customer of Docker Hub.

I will try out OpenUnison through the front door later, thank you for the pointers. I don't really feel great about my Keycloak deployment as I didn't know what I'm doing, and had to copy some example from their repo in order to make it run without a full OpenShift installation. This is a home lab so I don't really care that much, if it works it works, but as an example to show people and recommend, I can't suggest to users "hey you should cobble together Keycloak and oidc-proxy yourself, it's supported by nobody and you're totally on your own if something goes wrong." LOL

So OpenUnison looks like a nice alternative to that approach!

There is no database to manage or management UI to secure. Everything is K8s native through CRDs.

That's also a big advantage which should not be understated! Thanks very much for explaining in detail.

I will investigate the pull-through proxy option and see if there's a good automated solution for tagging new versions of the image from latest with a timestamp when they come in with a different SHA than before. I run my own Harbor registry already, so I'm pretty sure this option is (or at least ought to be) a fairly simple change to enable.

mlbiam commented 2 years ago

Using the latest tag will require people to use ImagePullPolicy: Always in order to be sure they are really pulling the latest version of the image – this is not advisable as you will quickly encounter Docker Hub rate limits, perhaps even as a paying customer of Docker Hub.

It's certainly a trade off. What we didn't want to happen was to keep old images that are no longer being updated published, or to remove images and have someone with an old manifest stuck in an image pull error.

mlbiam commented 2 years ago

closing do to inactivity

TremoloSecurity / kube-oidc-proxy

Migrating from jetstack to tremolosecurity failed #13