blog/devops/Istio-OIDC-Config/

[utterances-bot]

Configuring Istio with OIDC authentication - Homelab.blog

In this blog post, we will look at the first part of my ideal setup, which is to secure inbound communication via an authenticating reverse proxy (OAuth2_Proxy), and Keycloak.

https://homelab.blog/blog/devops/Istio-OIDC-Config/

[tillig]

Just tried to comment and thank you for this amazing article but it turns out there's an issue with the utterances app:

Error: utterances is not installed on Just-Insane/just-insane.github.io. If you own this repo, install the app. Read more about this change in the PR.

Anyway, this was great - I just got a similar setup done in Istio 1.6.4 (minor changes to the objects that need to be deployed) and was able to tweak it to lock down only one service rather than everything in the cluster (i.e., Gateway not using wildcards)... and only could have done it with this article. I'll have to blog my updates after I've gotten it tested a little more thoroughly, but I've spent like two weeks trying to nail this down and just got it totally working today. So, thanks.

[Just-Insane]

@tillig Glad I could help!

I've added the utterances app to this repo, so that should resolve that problem. Also not sure why I didn't get an email notification of this.

I'd be interested in checking out your blog if you got around to posting it!

[tillig]

I did end up posting it - here you go!

[Just-Insane]

Love the write-up! I'll be redeploying Istio at some point soon so the updated objects are very useful. The Istio Egress is also good to know about, I'd like to set that up at some point.

And yea, the Envoy filter headers are painful, they don't seem to be well documented, and I'm not sure about the security of what is set, but it does work at least. These articles may help in deciphering it though: https://medium.com/hepsiburadatech/istio-request-control-with-envoy-filters-request-headers-43dbd0dba7a9 and https://ultimatesecurity.pro/post/istio-secure-headers/

[flickerfly]

Why use the oidc provide for the oauth2-proxy rather than the keycloak provider?

[Just-Insane]

@flickerfly at the time of writing, the Keycloak provider had some issues (I forget the specifics at this point), however, the OIDC provider worked fine.

Feel free to give it a try with the Keycloak provider, I would think it works fine now. Might actually work better if it automates the auth renewal with the refresh token. I think that is what I am going to work on next.

[Just-Insane]

@flickerfly Looking into it, the keycloak provider is still limited more than the OIDC provider, please see this issue for more information: https://github.com/oauth2-proxy/oauth2-proxy/issues/956

[flickerfly]

Thanks!

[zufardhiyaulhaq]

Hi Justin, I want to ask something. currently I am using default behaviour with oauth2-proxy and Istio, just want to ask, is this statement is correct?

1. first time requests, go to Istio ingressgateway, got forwarded to oauth2-proxy service via external authz envoyfilter
2. oauth2-proxy check that authentication is missing, force client to to log in and redirected to the authentication provider
3. client login, login success, the oauth tokens are stored in the configured session store (cookie or redis in oauth2-proxy) and a cookie is set in the client
4. client get callback url
5. client do requests again, go to istio ingressgateway, forwarded to oauth2-proxy
6. oauth2-proxy verify the credentials, return 200 OK to Istio, Istio forward to upstream

also what is the different between default behaviour and your setup (I see you using some JWT verification in Istio), did the client side also get the cookies?

[Just-Insane]

@zufardhiyaulhaq Yes, that is the correct order of operations when using Istio and OAuth2_Proxy.

The purpose of the JWT verification in Istio is to ensure that the JWT (basically, the authentication token provided by your authentication provider) is actually from your authentication provider. It helps alleviate the potential that the authentication cookie is forged as well as ensures that no traffic is getting past the authentication flow.

[zufardhiyaulhaq]

@Just-Insane, I am not sure by the way, since the JWT already checked by oauth2-proxy, The problem with JWT validation by Istio is we cannot do that on the ingress side since Istio RequestAuthentication only can do that by selecting the destination pod label (I believe the configuration will apply in the sidecar), I have use cases that the service is not fronted by sidecar.

https://istio.io/latest/docs/reference/config/security/request_authentication/

[zufardhiyaulhaq]

I believe you also don't need to expose oauth2-proxy service via ingressgateway.

[flickerfly]

@zufardhiyaulhaq I can confirm that you don't need to expose oauth2-proxy via the ingressgateway if you are in the same cluster and have a svc setup in front of oauth2-proxy. If you want to support multi-cluster scenarios, that'll probably require ingressgateway or something similar to expose the required svc outside the cluster.

[patrickleet]

Might actually work better if it automates the auth renewal with the refresh token.

This is what I'm struggling with now.

[patrickleet]

actually, that was pretty straightforward:

- --cookie-refresh=4m0s

[tzakrajs]

How can I have a single ingressgateway with some VirtualServices or Gateways unauthenticated while the rest are unauthenticated? The issue seems to be that the EnvoyFilter applies to the entire ingressgateway and even when I try to scope the RequestAuthentication/AuthorizationPolicy/EnvoyFilter to specific pods with labels I just get 403 RBAC: access denied when I hit any VirtualService until I reapply the RequestAuthentication/AuthorizationPolicy/EnvoyFilter to select for app: istio-ingressgateway.

Thanks for any help.

[tzakrajs]

while the rest are authenticated*

[flickerfly]

Probably depends a lot on your version of Istio/Envoy, but you can see in Travis' work he filters by SNI. https://www.paraesthesia.com/archive/2020/09/03/setting-up-oauth2-proxy-with-istio/

[patrickleet]

@flickerfly he's using multiple gateways too, or needs to if the example were to cover more services..

The SNI will match a whole gateway. So for example having *.example.com and dashboard.example.com in one gateway, and matching the SNI dashboard.example.com will also apply to *.example.com. At least one exact match is required but applied to whole gateway.

Furthermore certs can't be shared between multiple gateways without getting some 404 errors, so you need to be careful about wildcard certs as well.

With those considerations you likely need multiple certs and multiple gateways.

I have an example repo with istio/keycloak set up now here: https://github.com/CloudNativeEntrepreneur/jx3-gitops-template

@tzakrajs

Check out helmfiles/auth for keycloak set up with operator, and helmfiles/istio-system for envoyfilter.

It's authenticating a dashboard from the CI/CD tool JX, so check out those istio related configs too in helmfiles/jx.

It's also connected with external-dns and cert-manager, and has an knative set up, which can also be auth'd using EnvoyFilters.

[tzakrajs]

@patrickleet thanks for your help

So its not pretty but basically I am now adding a new configPatch for each SNI that I want to apply Authentication to:

- applyTo: HTTP_FILTER
  match:
    context: GATEWAY
    # context: SIDECAR_INBOUND
    listener: 
      filterChain:
        sni: hostname
        filter:
          name: "envoy.http_connection_manager"
          # subFilter:
          #   name: "envoy.router"
  patch:
    operation: INSERT_BEFORE
    value:
      name: envoy.filters.http.ext_authz
      typed_config:
        "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
        http_service:
          server_uri:
            uri: http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180
            cluster: outbound|4180||oauth2-proxy.oauth2-proxy.svc.cluster.local
            timeout: 10s
          authorizationRequest:
            allowedHeaders:
              patterns:
              - exact: cookie
          authorizationResponse:
            allowedUpstreamHeaders:
              patterns:
              - exact: authorization

I have several of these but it works fine and this isn't for enterprise, so I think I am good for now.