ldap inserting http schema into okta redirect_uri

[maks-mikhalov]

Hi guys, I'm trying to implement k8s<->okta authentication but getting into a weird issue. I'm opening the dashboard URL it redirects to /login/ldap and to okta after that. But okta throwing this error: Identity Provider: Unknown Error Code: invalid_request Description: The 'redirect_uri' parameter must be an absolute URI that is whitelisted in the client app settings. I've checked rediret_uri and it set to dashboard uri but schema set to HTTP instead of HTTPS. Could you tell me where I could define schema for return uri? Here are my configs: Cluster: EKS 1.16 Ingress: Istio 1.6.5 Openunison-k8s-login-oidc: 1.0.19 OIDC provider: okta values.yaml:

  openunison_host: "dev-eks-test-01.testdomain.com"
  dashboard_host: "dev-eks-test-01.testdomain.com"
  api_server_host: "dev-eks-test-01.testdomain.com"
  session_inactivity_timeout_seconds: 900
  k8s_url: "https://bububu.lala.us-east-1.eks.amazonaws.com"

cert_template:
  ou: "dev-eks-test-01"
  o: "whereami"
  l: "dev-eks-test-01"
  st: "TX"
  c: "US"

image: "31415926.dkr.ecr.us-west-2.amazonaws.com/openunison-k8s-login-oidc:latest"
myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: dev-eks-test-01
enable_impersonation: false

dashboard:
  namespace: "kubernetes-dashboard"
  cert_name: "kubernetes-dashboard-certs"
  label: "k8s-app=kubernetes-dashboard"
  service_name: kubernetes-dashboard
certs:
  use_k8s_cm: false

trusted_certs: []

monitoring:
  prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s

oidc:
  client_id: omgid
  auth_url: https://whereami-test.okta.com/oauth2/v1/authorize
  token_url: https://whereami-test.okta.com/oauth2/v1/token
  user_in_idtoken: false
  userinfo_url: https://whereami-test.okta.com/oauth2/v1/userinfo
  domain: "whereami-test"
  scopes: openid email profile groups
  claims:
    sub: sub
    email: email
    given_name: given_name
    family_name: family_name
    display_name: name
    groups: groups

[mlbiam]

The protocol is set based on the incoming URL. Sounds like between the ingress and the pod is HTTP, not HTTPS. How is your Ingress configured?

[mlbiam]

Also, the dashboard_host, openunison_host and api_server_host need to be different. i realiaze those are fake host names but just wanted to verify you have them as different in your real config

[maks-mikhalov]

@mlbiam thanks for the fast answer. Yeah, they all fake hosts. All three hosts are different. I also thought that protocol is taken from the incoming request, but I'm sure that x-forwarded-proto and x-forwarded-port are delivered to the host. I also hardcoded headers just in case. Here is ingress configuration:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: oidc-client
spec:
  hosts:
    - dev-eks-test-01.testdomain.com
    - dev-eks-test-01db.testdomain.com
    - dev-eks-test-01api.testdomain.com
  gateways:
    - oidc-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: openunison-orchestra
        port:
          number: 80
      headers:
        response:
          add:
            x-forwarded-proto: https
            x-forwarded-port: "443"

[mlbiam]

Looks like the X-Forwarded-Proto header isn't being honored when generating the redirect. I don't have Itsio running at the moment but give the image tremolosecurity/betas:k8s-login-oidc-1.0.20 a try. I think the X-Forwarded-Proto needs to be in the request section, not the response to add the header to the inbound request? Finally, the header should use proper case so it matches up instead of lower case.

Let me know if this works.

Thanks

[mlbiam]

Some good news, I was able to verify this build with istio. (The build in the logs should say 1.0.20-2020071905). I'm trying to workout how to go from the VirtualService --> OpenUnison over https and using a destinationroute to setup session stickyness so HA is supported.

Also, you're original VirtualService works correctly as is, i was wrong about my suggested changes.

[mlbiam]

Got this working with end-to-end TLS and not needing the proto headers. Create a DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: openunison-tls-rule
  namespace: openunison
spec:
  host: openunison-orchestra
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: oustickysession
          path: /
          ttl: 0s
    tls:
      mode: SIMPLE

Then update your VirtualService :

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: oidc-client
spec:
  hosts:
    - dev-eks-test-01.testdomain.com
    - dev-eks-test-01db.testdomain.com
    - dev-eks-test-01api.testdomain.com
  gateways:
    - oidc-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: openunison-orchestra
        port:
          number: 443

you can also go back to the GA build of OpenUnison (docker.io/tremolosecurity/openunison-k8s-login-oidc:latest) in orchestra object (kubectl edit openunison orchestra -n openunison)

[maks-mikhalov]

Thanks a lot, moving from 80 to 443 port number in virtual service resolved the issue. But faced with another one. Looks like OpenUnison is generating self-signed certificate and inserting it as idp-certificate-authority-data. But we're doing SSL termination at ingress. So when we're doing kubectl get pods we're getting: Unable to connect to the server: Get "https://dev-eks-test-01.testdomain.com/auth/idp/k8sIdp/.well-known/openid-configuration": x509: certificate signed by unknown authority Is there any possibility to replace this cert with one that we're using with ingress? Thanks

[mlbiam]

Your ingress cert, is it self signed or commercially signed?

[maks-mikhalov]

It is commercially signed

[mlbiam]

Updated our troubleshooting guide for this use case, sorry thought it was already in there: https://github.com/TremoloSecurity/OpenUnison/wiki/troubleshooting#using-a-commercially-signed-ingress-certificate

[maks-mikhalov]

Thanks a lot. That was helpful. Sorry for being so problematic, but there is a new issue now. Sorry for being so problematic :-)

root@4616fb37e03a:/tmp# kubectl get pods
I0721 13:29:25.792246   73309 round_trippers.go:420] GET https://31415926.yl4.us-north-1.eks.amazonaws.com/api/v1/namespaces/openunison/pods?limit=500
I0721 13:29:25.792261   73309 round_trippers.go:427] Request Headers:
I0721 13:29:25.792266   73309 round_trippers.go:431]     Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
I0721 13:29:25.792271   73309 round_trippers.go:431]     User-Agent: kubectl/v1.18.2 (darwin/amd64) kubernetes/52c56ce
I0721 13:29:25.832802   73309 round_trippers.go:446] Response Status: 401 Unauthorized in 40 milliseconds
I0721 13:29:25.832830   73309 round_trippers.go:449] Response Headers:
I0721 13:29:25.832839   73309 round_trippers.go:452]     Audit-Id: 523db02c-faf4-4fe4-a917-a4c1153731d7
I0721 13:29:25.832848   73309 round_trippers.go:452]     Cache-Control: no-cache, private
I0721 13:29:25.832855   73309 round_trippers.go:452]     Content-Type: application/json
I0721 13:29:25.832862   73309 round_trippers.go:452]     Content-Length: 129
I0721 13:29:25.832868   73309 round_trippers.go:452]     Date: Tue, 21 Jul 2020 18:29:25 GMT
I0721 13:29:25.832896   73309 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0721 13:29:25.833919   73309 helpers.go:216] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}]
F0721 13:29:25.833963   73309 helpers.go:115] error: You must be logged in to the server (Unauthorized)

I've created k8s-admins group in okta, assigned myself there. Also tried to add Everyone group into Clusterrolebinding with the same effect. I don't see anything in the orchestra logs

[maks-mikhalov]

Also after some time, I'm starting getting this:

kubectl get pods
Unable to connect to the server: Get "https://dev-eks-test-01.testdomain.com/auth/idp/k8sIdp/.well-known/openid-configuration": x509: certificate signed by unknown authority

Not sure if it is related to previous issue

[mlbiam]

it looks like its trying to talk directly to your API server instead of openunison. is enable_impersonation set to true in your values.yaml?

[maks-mikhalov]

enable_impersonation is set to false

[mlbiam]

For eks it needs to be set to true because EKS doesn't support openid connect

[maks-mikhalov]

Thanks a lot, looks like there is the last issue :-)

root@49629ba1f825:/tmp# kubectl get pods -n openunison
Error from server (Forbidden): pods is forbidden: User "00ul0lnepTSTlXfny4x6" cannot list resource "pods" in API group "" in the namespace "openunison"

Here is rolebinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: okta-cluster-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: k8s-admins
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: Everyone

Any ideas why it is getting user instead of the group?

[mlbiam]

When you login to openunison, click on your username in the upper left (00ul0lnepTSTlXfny4x6). It should show a list of groups. Do you see Everyone and k8s-admins?

[maks-mikhalov]

It is not showing groups, just:

00ul0lnepTSTlXfny4x6's Profile
Attributes
Login ID00ul0lnepTSTlXfny4x6
Roles
00ul0lnepTSTlXfny4x6 has no roles assigned

And my user is definitely member of both

[mlbiam]

In your okta configuration, are you filtering group claims?

[maks-mikhalov]

That was it. So it is working like a charm now :-) Just need to figure it out how to deal with multiple clusters with a single authentication

[mlbiam]

Awesome! we're working on a multi-cluster portal so stay tuned there. What out customers are currently doing with multi cluster is to use our kubectl login plugin to switch between clusters. So if you are logged in to Okta running the login command will pop up a new browser tab but won't prompt you to login again.

[maks-mikhalov]

Sounds cool. I've already installed the kubectl plugin and it is working well.

[maks-mikhalov]

Also, small question - does orchestra know how to refresh the okta session or it is user responsibility?

[mlbiam]

orchestra doesn't refresh the okta session. Once authenticated, orchestra maintains its own session as an oidc-sessions object in the openunison namespace. If you need to revoke a session you would delete the objects associated with the user in question and when their id_token expires it will fail to refresh.

[mlbiam]

closing since it seems this issue has been solved