OpenID Connect to AWS eks

[brackend]

Hi,

I see there is a very good guide for a eks configuration using active directory https://www.tremolosecurity.com/post/multi-tenant-amazon-eks-the-easy-way-part-i-authentication.

I'm working on deploying using an openid token in OICD flow for AWS EKS. Expecting that a configuration will allow me to do userinfo lookup and retrieve groups from the IDP. Which in turn map to k8s aws-auth roles.

1st: should this work? 2nd: is there a guide similar to the active directory guide mentioned above? 3rd: how do I map say "memberOf"(pingid) to "groups" claim when interrogating userinfo endpoint

Thank you!

[mlbiam]

Which in turn map to k8s aws-auth roles.

Not exactly. The aws-auth configmap in the kube-system namespace is used by the aws token validator to map your IAM role to a Kubernetes RBAC Group or User. When using OpenUnison, your aws IAM role no longer applies. Your groups from your idp are mapped to directly from your idp.

2nd: is there a guide similar to the active directory guide mentioned above?

What's your IdP? You would follow the guide in this repo's README.md but just set impersonation to true.

3rd: how do I map say "memberOf"(pingid) to "groups" claim when interrogating userinfo endpoint

Are you saying that your userinfo endpoint includes a memberOf claim and that's what you want to use for your groups? If so set oidc.claims.groups=memberOf in your helm chart's values.yaml

[brackend]

1st: Ok got you. 2nd: ping identity is the IDP. I've followed the readme. Probably missed something but see below... 3rd: Thanks will try it. pingid responds with "memberOf" instead of group. Hence need to map it to group.

Getting this error: [2021-08-26 19:42:51,745][XNIO-1 task-1] ERROR OpenIDConnectAuthMech - Could not retrieve token : 401 / Unauthorized [2021-08-26 19:42:51,746][XNIO-1 task-1] INFO AccessLog - [AuFail] - scale - https://ou./auth/oidc - cn=none - enterprise_idp [100.64.10.117] - [f04b6d17e686cc0ebb7024fff159326bdaafddaf1] [2021-08-26 19:42:52,280][XNIO-1 task-1] INFO AccessLog - [Error] - scale - https://ou./auth/oidc - uid=Anonymous,o=Tremolo - NONE [100.64.10.117] - [f04b6d17e686cc0ebb7024fff159326bdaafddaf1] [2021-08-26 19:42:52,280][XNIO-1 task-1] ERROR ConfigSys - Could not process request javax.servlet.ServletException: Could not load user data at com.tremolosecurity.unison.proxy.auth.openidconnect.OpenIDConnectAuthMech.doGet(OpenIDConnectAuthMech.java:248) ~[unison-auth-openidconnect-1.0.23-1.jar:?] ...

And in terms of http requests/responses: when first accessing ou. ( k8sou.apps.domain.int) it redirects to /login/ldap, which then in turn redirects to our IDP for authentication but includes max-age=0 in the query string. I'm prompted twice to authenticate and finally fails for max-age=0:

https://ou./auth/oidc?error_description=Unable to accommodate the requested maximum authentication age - the requested max_age was 0 while the actual age is 16 (in seconds)&state=security_token=&error=access_denied#

[brackend]

Thank you for help above. Any help with this log issue? Do you need more info or clarity on the issue?

I've tried to increase the log level by creating a ConfigMap setting loglevel=debug. But hasn't added anything new to the orchestra log

[mlbiam]

i'm sorry. OpenIDConnectAuthMech - Could not retrieve token : 401 / Unauthorized usually means your combination of clientid and client secret are being rejected by your idp (ping). if you take a look at the logs there you'll probably see the issue. double check your client id and secret

[brackend]

Sorry just came back to oidc again. It works fine when I remove max-age=0 from the end of the request generated by openunison to the idp. I'm not sure where max-age value is set or how it's set. https://idp/as/authorization.oauth2?client_id=whatever&response_type=code&scope=openid&redirect&state=security_token&max_age=0

Do you know where it’s getting max age of 0 from?

[brackend]

Hi Marc, @mlbiam,

could you please take a look? it looks like it might be a simple fix. It works fine if I remove max_age=0 manually from the redirect url.

[mlbiam]

Sorry for the delay. That is odd. The docs from Ping say this is a valid parameter (https://docs.pingidentity.com/bundle/pingfederate-103/page/nfr1564003024683.html). Is there anything in the Ping logs when you try to login?

max-age=0 forces the idp (ping) to re-authenticate you. It's a standard part of the openid connect protocol. I did some searching and that error string Unable to accommodate the requested maximum authentication age doesn't appear anywhere on the internet or in ping's resources. I reached out to some colleagues to see what they think.

I'll make this configurable. Expect something from me in a couple of days.

[mlbiam]

@brackend what version of ping are you running? and what is the actual module that is doing the authentication? (i've got some folks from ping helping me out)

[mlbiam]

got confirmation from my friends at Ping the best way is to remove max_age. They're saying its likely an interaction between MFA adapters on Ping that aren't working together properly. Stay tuned. Should hear from me tomorrow.

[brackend]

Super thank you for that. I take it you no longer need version id and module.

[mlbiam]

@brackend sorry it took longer to get this working. This repo is now deprecated, so new features are being added to https://github.com/OpenUnison/openunison-k8s, but you can re-use your existing secret and values.yaml to upgrade. To enabled the new configuration option:

1. Follow the instructions https://openunison.github.io/upgrading/
2. At this point you're in the same spot, but you're running off the right repos
3. Add the betas repo to your local deployment:

helm repo add tremolo-betas https://nexus.tremolo.io/repository/helm-betas/
helm repo updates

4. Add oidc.forceauthentication: false to your values.yaml
5. Change image to docker.io/tremolosecurity/betas:openunison-k8s-1025

update your orchestra-login-portal deployment:

helm upgrade orchestra-login-portal tremolo-betas/orchestra-login-portal --namespace openunison -f ~/Documents/Documents/projects/kube-oidc-proxy-values.yaml 

Now when you try to login max_age should no longer be in the request url

[brackend]

@mlbiam got passed max age - thanks for this. Redirects & oidc flows work. User is created with all the correct groups like so:

apiVersion: openunison.tremolo.io/v1 kind: User metadata: creationTimestamp: "2022-01-15T11:08:00Z" generation: 1 name: xx-xx-xxxx-xx-xxx namespace: openunison resourceVersion: "" uid: dddddddd-dddd-dddd-dddd-dddddddddddd spec: email: Dara.Bracken@mail.com first_name: mysub groups: - A-GROUP - B-GROUP last_name: Dara sub: mysub uid: xx-xx-xxxx-xx-xxx

Seeing this now.

[2022-01-16 21:01:49,510][XNIO-1 task-1] INFO AccessLog - SRCH op=2 con=1 base='o=Tremolo' filter='(uid=)' scope='2' attribs='' [2022-01-16 21:01:49,512][XNIO-1 task-1] INFO AccessLog - RESULT op=2 con=1 result=0 time=3 [2022-01-16 21:01:49,518][XNIO-1 task-1] INFO AccessLog - SRCH-RESULT op=2 con=1 entries=0 time=9 [2022-01-16 21:01:49,521][XNIO-1 task-1] ERROR JITAuthMech - Could not execute workflow 'jitdb' on 'sub=mysub,ou=oidc,o=Tremolo'com.tremolosecurity.provisioning.core.ProvisioningException: Could not reload user at com.tremolosecurity.provisioning.core.WorkflowImpl.executeWorkflow(WorkflowImpl.java:601)

At an earlier point in the log also saw this:

[2022-01-16 20:58:50,353][main] INFO ScaleMain - Display Name Attribute Name: 'sub' [2022-01-16 20:58:50,353][main] INFO ScaleMain - Front Page Title: 'Kubernetes Access Portal' [2022-01-16 20:58:50,353][main] INFO ScaleMain - Front Page Text: 'Use this portal to create and access namespaces in Kubernetes' [2022-01-16 20:58:50,353][main] INFO ScaleMain - User Fields Editable: 'false' [2022-01-16 20:58:50,353][main] INFO ScaleMain - Save User Workflow: '' [2022-01-16 20:58:50,353][main] INFO ScaleMain - User ID Attribute Name: 'uid' [2022-01-16 20:58:50,353][main] INFO ScaleMain - Show Portal Orgs: 'false' [2022-01-16 20:58:50,353][main] INFO ScaleMain - Logout URL: '/logout' [2022-01-16 20:58:50,353][main] INFO ScaleMain - Warn when number of minutes left in the user's session: '5' [2022-01-16 20:58:50,353][main] INFO ScaleMain - canDelegate: 'NO' [2022-01-16 20:58:50,354][main] INFO ScaleMain - canPreApprove: 'NO' [2022-01-16 20:58:50,354][main] INFO ScaleMain - enableApprovals: 'false' [2022-01-16 20:58:50,354][main] INFO ScaleMain - Role Attribute Name: 'groups' [2022-01-16 20:58:50,354][main] INFO ScaleMain - sub Display Name: 'Login ID' [2022-01-16 20:58:50,354][main] INFO ScaleMain - sub Read Only: 'true' [2022-01-16 20:58:50,354][main] WARN ScaleMain - sub Required not found [2022-01-16 20:58:50,354][main] WARN ScaleMain - sub Reg Ex not found [2022-01-16 20:58:50,354][main] WARN ScaleMain - sub Reg Ex Failed Message not found [2022-01-16 20:58:50,354][main] WARN ScaleMain - sub Minimum Characters not found [2022-01-16 20:58:50,354][main] WARN ScaleMain - sub Maximum Characters not found [2022-01-16 20:58:50,354][main] WARN ScaleMain - sub Attribute Type not found [2022-01-16 20:58:50,355][main] ERROR UnisonConfigManagerImpl - Could not initialize filter java.lang.Exception: Approval attribute names not found at com.tremolosecurity.scalejs.ws.ScaleMain.initFilter(ScaleMain.java:1543) ~[unison-scalejs-main-1.0.25.jar:?] at com.tremolosecurity.config.util.UrlHolder.init(UrlHolder.java:175) ~[unison-sdk-1.0.25.jar:?]

Tried: server.shadowUsers.api.config.alwaysMapUIDInFilter=false ( was true)

[mlbiam]

[2022-01-16 21:01:49,510][XNIO-1 task-1] INFO AccessLog - SRCH op=2 con=1 base='o=Tremolo' filter='(uid=)' scope='2' attribs='' [2022-01-16 21:01:49,512][XNIO-1 task-1] INFO AccessLog - RESULT op=2 con=1 result=0 time=3 [2022-01-16 21:01:49,518][XNIO-1 task-1] INFO AccessLog - SRCH-RESULT op=2 con=1 entries=0 time=9 [2022-01-16 21:01:49,521][XNIO-1 task-1] ERROR JITAuthMech - Could not execute workflow 'jitdb' on 'sub=mysub,ou=oidc,o=Tremolo'com.tremolosecurity.provisioning.core.ProvisioningException: Could not reload user at com.tremolosecurity.provisioning.core.WorkflowImpl.executeWorkflow(WorkflowImpl.java:601)

this usually happens because there's an attribute missing. but based on your user object it looks like everything is there. Can you double check against the claims in your JWT?

[2022-01-16 20:58:50,355][main] ERROR UnisonConfigManagerImpl - Could not initialize filter java.lang.Exception: Approval attribute names not found at com.tremolosecurity.scalejs.ws.ScaleMain.initFilter(ScaleMain.java:1543) ~[unison-scalejs-main-1.0.25.jar:?] at com.tremolosecurity.config.util.UrlHolder.init(UrlHolder.java:175) ~[unison-sdk-1.0.25.jar:?]

You can ignore this

Tried: server.shadowUsers.api.config.alwaysMapUIDInFilter=false ( was true)

Are you using a custom myvd configuration? If so, can you post the contents of the configmap?

[brackend]

Not usre where to see the JWT. It's "auth code flow". No changes to myvd. Just tried temporarily change that one parameter for uid. But here it is: myvd.conf: >-

Global AuthMechConfig

server.globalChain=accesslog
server.globalChain.accesslog.className=com.tremolosecurity.proxy.myvd.log.AccessLog
server.nameSpaces=rootdse,myvdroot,shadowUsers
server.rootdse.chain=dse
server.rootdse.nameSpace=
server.rootdse.weight=0
server.rootdse.dse.className=net.sourceforge.myvd.inserts.RootDSE

server.rootdse.dse.config.namingContexts=o=Tremolo server.myvdroot.chain=root server.myvdroot.nameSpace=o=Tremolo server.myvdroot.weight=0 server.myvdroot.root.className=net.sourceforge.myvd.inserts.RootObject server.shadowUsers.chain=mapping,api server.shadowUsers.nameSpace=ou=shadow,o=Tremolo server.shadowUsers.weight=0 server.shadowUsers.enabled=true server.shadowUsers.debug.className=net.sourceforge.myvd.inserts.DumpTransaction server.shadowUsers.debug.config.logLevel=info server.shadowUsers.debug.config.label=k8s server.shadowUsers.mapping.className=net.sourceforge.myvd.inserts.mapping.AttributeMapper server.shadowUsers.mapping.config.mapping=mail=email,givenName=first_name,sn=last_name serer.shadowUsers.api.className=com.tremolosecurity.myvd.K8sCrdInsert server.shadowUsers.api.config.alwaysMapUIDInFilter=true server.shadowUsers.api.config.nameSpace=openunison server.shadowUsers.api.config.k8sTargetName=k8s server.shadowUsers.api.config.alwaysMapUIDInFilter=true

[brackend]

Seems odd that uid change from value: xx-11-xxx-11-xxx on line 8 to value: 1111111(sub) on line 14. Also, I don't see group":"openunison.tremolo.io ( line 1) associated within the user object.

1. [2022-01-17 20:48:57,805][XNIO-1 task-1] WARN OpenShiftTarget - Unexpected result calling 'https://172.20.0.1:443/apis/openunison.tremolo.io/v1/namespaces/openunison/users/xx-11-xxx-11-xx' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"users.openunison.tremolo.io \"xx-11-xxx-11-xx\" not found","reason":"NotFound","details":{"name":"xx-11-xxx-11-xx","group":"openunison.tremolo.io","kind":"users"},"code":404}
3. [2022-01-17 20:48:57,834][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=true Add user=1111111 workflow=jitdb approval=0 sub='1111111'
4. [2022-01-17 20:48:57,834][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 sub='1111111'
5. [2022-01-17 20:48:57,834][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 first_name='1111111'
6. [2022-01-17 20:48:57,834][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 last_name='nnnnn'
7. [2022-01-17 20:48:57,835][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 email='nnnnn.sssssssssss@firstdata.com'
8. [2022-01-17 20:48:57,835][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 uid='xx-11-xxx-11-xx'
9. [2022-01-17 20:48:57,835][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 group='GROUP1'
10. .....
11. [2022-01-17 20:48:57,838][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 group='GROUP11'
12. [2022-01-17 20:48:57,838][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 group='GROUP1111'
13. [2022-01-17 20:48:57,838][XNIO-1 task-1] INFO ProvisioningEngineImpl - target=jitdb entry=false Add user=1111111 workflow=jitdb approval=0 group='GROUP11111'
14. [2022-01-17 20:48:57,860][XNIO-1 task-1] INFO AccessLog - SRCH op=2 con=1 base='o=Tremolo' filter='(uid=1111111)' scope='2' attribs=''
15. [2022-01-17 20:48:57,862][XNIO-1 task-1] INFO AccessLog - RESULT op=2 con=1 result=0 time=3
16. [2022-01-17 20:48:57,868][XNIO-1 task-1] INFO AccessLog - SRCH-RESULT op=2 con=1 entries=0 time=9
17. [2022-01-17 20:48:57,871][XNIO-1 task-1] ERROR JITAuthMech - Could not execute workflow 'jitdb' on 'sub=1111111,ou=oidc,o=Tremolo'com.tremolosecurity.provisioning.core.ProvisioningException: Could not reload user
18. at com.tremolosecurity.provisioning.core.WorkflowImpl.executeWorkflow(WorkflowImpl.java:601)

[mlbiam]

server.shadowUsers.api.config.alwaysMapUIDInFilter=true

this is where it is now, right?

What does the oidc.claims section of your values.yaml look like? Do you have sub: sub? The switch usually happens if you're mapping the sub to a different attribute and its not consistent.

[brackend]

Yes, set to true and yes sub: sub is set in claims

[mlbiam]

What is oidc.user_in_idtoken set to? Try flipping it and update orchestra-login-portal.

[brackend]

It was set to false. The idtoken is minimal in this case. user-info endpoint is used to get the information.
But tried flipping to see what happens. User object only gets sub as expected and results in same error.

The user object seemed ok, at least to me the first time around (shown above somewhere).

[mlbiam]

Pretty common. Odd, OK, I'll add something to make this easier to debug.

[brackend]

Just noticed that I was not using beta crds. the user object is different.

[mlbiam]

Can you confirm your helm chart versions?

[brackend]

• name: openunison-operator version: 2.0.4 repository: https://
• name: orchestra version: 2.2.0 repository: https://
• name: orchestra-login-portal version: 2.1.1 repository: https://

[mlbiam]

Those versions are correct. I added a way to set the debug logs more easily. First, update the beta-repos on your system:

helm repo update

then, create the below ConfigMap:

apiVersion: v1
data:
  log4j2.xml: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<Configuration>\r\n  <Appenders>\r\n
    \   <Console name=\"STDOUT\" target=\"SYSTEM_OUT\">\r\n      <PatternLayout pattern=\"[%d][%t]
    %-5p %c{1} - %m%n\"/>\r\n    </Console>\r\n  </Appenders>\r\n  <Loggers>\r\n\r\n
    \   <Root level=\"info\">\r\n      <AppenderRef ref=\"STDOUT\"/>\r\n    </Root>
    \r\n\r\n    <Logger name=\"com.tremolosecurity.myvd.K8sCrdInsert\"  level=\"debug\">\r\n
    \     <AppenderRef ref=\"STDOUT\" level=\"debug\"/>\r\n    </Logger>\r\n  </Loggers>\r\n</Configuration>\r\n"
kind: ConfigMap
metadata:
  name: oudebug
  namespace: openunison

Next, set openunison.debugConfigMap: oudebg in your values.yaml. Then update the orchestra deployment:

helm upgrade orchestra tremolo-betas/orchestra --namespace openunison -f /path/to/values.yaml

Once the pod restarts, try logging in. I'm going to walk through input data so we can see where the disconnect is.

First, here are my claims:

{
  "iss": "https://k8sou.212-2-246-6.nip.io/auth/idp/k8sIdp",
  "aud": "kubernetes",
  "exp": 1642471812,
  "jti": "oFjZLiz1eYkOEAhu6YvQFQ",
  "iat": 1642471752,
  "nbf": 1642471632,
  "sub": "00u3fusfj6jFLURbp357",
  "name": " Mosley",
  "groups": [
    "demo-k8s",
    "Everyone",
    "k8s-users",
    "k8s-admins"
  ],
  "preferred_username": "x-48-xx-48-xux-51-xfusfjx-54-xjflurbpx-51-xx-53-xx-55-x",
  "email": "marc+mmosley@tremolo.io"
}

The important one is sub. When I login, the following User object gets created:

apiVersion: openunison.tremolo.io/v1
kind: User
metadata:
  name: x-48-xx-48-xux-51-xfusfjx-54-xjflurbpx-51-xx-53-xx-55-x
  namespace: openunison
spec:
  email: marc+mmosley@tremolo.io
  first_name: Matt
  groups:
  - demo-k8s
  - Everyone
  - k8s-users
  - k8s-admins
  last_name: Mosley
  sub: 00u3fusfj6jFLURbp357
  uid: x-48-xx-48-xux-51-xfusfjx-54-xjflurbpx-51-xx-53-xx-55-x

the spec.uid and metadata.name should match. If they don't, what's the difference? Next, look in OpenUnison's logs. Look for :

[2022-01-18 02:09:12,602][XNIO-1 task-1] INFO  AccessLog - SRCH op=2 con=1 base='o=Tremolo' filter='(uid=00u3fusfj6jFLURbp357)' scope='2' attribs=''
[2022-01-18 02:09:12,605][XNIO-1 task-1] DEBUG K8sCrdInsert - orirignal filter : '(uid=00u3fusfj6jFLURbp357)'
[2022-01-18 02:09:12,605][XNIO-1 task-1] DEBUG K8sCrdInsert - orirignal filter : '(uid=00u3fusfj6jFLURbp357)'
[2022-01-18 02:09:12,606][XNIO-1 task-1] DEBUG K8sCrdInsert - Looking up user 'x-48-xx-48-xux-51-xfusfjx-54-xjflurbpx-51-xx-53-xx-55-x' in namespace 'openunison'
[2022-01-18 02:09:12,606][XNIO-1 task-1] DEBUG K8sCrdInsert - Looking up user 'x-48-xx-48-xux-51-xfusfjx-54-xjflurbpx-51-xx-53-xx-55-x' in namespace 'openunison'
[2022-01-18 02:09:12,637][XNIO-1 task-1] INFO  AccessLog - RESULT op=2 con=1 result=0 time=36

The original filter line has a uid that should match the original sub. The Looking up user line should match the User object's metadata.name and spec.uid. Does it?

[brackend]

Hi Marc @mlbiam , There was typo in the orchestra yaml and array was not quite right

<removed this section - user error :( >

So now seems to have mounted correctly. but.. I'm not seeing debug entries in log file. Not sure how to enable debug.

BTW: this looks ok apiVersion: openunison.tremolo.io/v1 kind: User metadata: creationTimestamp: "2022-01-19T18:58:57Z" generation: 1 name: xx-11-xxxx-11-xxxxx namespace: openunison resourceVersion: "xxxxxxxxx" uid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx spec: email: Dara@mail.com first_name: SUBHERE groups:

• GROUP1
• GROUP11
• GROUP111 last_name: Dara sub: subhere uid: xx-11-xxxx-11-xxxxx

For diagnostic purposes I can update the yamls directly in the environment.

[brackend]

Sorry just another note. I had not been using the latest deployment.yaml file ( for some reason I had an old file in the template directory ). But when I used the latest one I get this error: Error: Unable to access jarfile /usr/local/openunison/javascript-operator.jar. It's also referencing the image: tremolosecurity/betas:openunison-k8s-1025; as is orchestra.

[mlbiam]

Sorry just another note. I had not been using the latest deployment.yaml file ( for some reason I had an old file in the template directory ). But when I used the latest one I get this error: Error: Unable to access jarfile /usr/local/openunison/javascript-operator.jar.

did you change the operator instead of the orchestra deployment? I'm wondering if it's best to start over? Since you have the values.yaml and your source Secret, everything else should be OK. I would remove all your helm deployments, so helm list shows an empty list. Then install the operator:

helm repo update
helm install openunison tremolo/openunison-operator --namespace openunison

Once that's deployed, deploy the beta orchestra using your values.yaml:

helm install orchestra tremolo-betas/orchestra --namespace openunison -f /path/to/values.yaml

once that's running, and ready (1/1), deploy the openunison portal:

helm install orchestra-login-portal tremolo-betas/orchestra-login-portal --namespace openunison -f /path/to/values.yaml

based on the object you have above everything should be lining up

[brackend]

This is working now. Thanks for all the help.
As in the comment above, see: https://github.com/OpenUnison/openunison-k8s-login-activedirectory/issues/109