AKHQ Login via OIDC Keycloak not working with 0.17 or 0.18

[wilkej]

Hello AKHQ Team,

thank you for the project. Today I tested to update our AKHQ setup from 0.16 to 0.17. After the update I'm not able to login anymore via OIDC.

We use Keycloak as OIDC provider and with keycloak we create a token which includes the AKHQ roles which should be allowed and the topic filter regex. An example token:

{
  "exp": "...",
  "iat": "...",
  "jti": "...",
  "iss": "OIDC-URL",
  "sub": "...",
  "typ": "Bearer",
  "azp": "client-ID",
  "session_state": "...",
  "acr": "1",
  "allowed-origins": [],
  "resource_access": {
    "client-id": {
      "roles": [
        "topic/read",
        "group/read",
        "topic/data/read"
      ]
    }
  },
  "scope": "openid profile email",
  "topics-filter-regexp": [
    "^my_test_latest_r-type_v1$",
    "^my_test_latest_a_v1$"
  ],
  "email_verified": false,
  "roles": [
    "topic/read",
    "group/read",
    "topic/data/read"
  ],
  "name": "Firstname Lastname",
  "preferred_username": "user@domain",
  "given_name": "Firstname",
  "locale": "en",
  "family_name": "Lastname",
  "email": "firstname.lastname@domain.com"
}

This is the AKHQ config

  # Auth & Roles (optional)
  security:
    default-group: no-roles # Default groups for all the user even unlogged user
    oidc:
      enabled: true
      providers:
        keycloak:
          label: "Login with Desti Passport"
          username-field: preferred_username # default in jwt token claim, preferred_username 
          groups-field: roles
          default-group: no-roles

Here is the token create by AKHQ log output from AKHQ 0.16.0

Attributes: 
at_hash=>..., 
sub=>..., 
email_verified=>false, 
roles=>["topic\/read","group\/read","topic\/data\/read"], 
iss=>akhq, 
typ=>ID, 
preferred_username=>..., 
given_name=>..., 
locale=>en, 
acr=>0, 
topics-filter-regexp=>["^my_test_latest_r-type_v1$","^my_test_latest_a_v1$"], 
nbf=>Wed Sep 22 08:23:51 UTC 2021, 
azp=>client-id, 
auth_time=>..., 
name=>Firstname Lastname, 
exp=>Wed Sep 22 09:23:51 UTC 2021, 
session_state=>..., 
iat=>Wed Sep 22 08:23:51 UTC 2021, 
family_name=>Lastname, 
email=>firstname.lastname@domain.com, 
oauth2Provider=>keycloak, 
username=>...

and what I get when using AKHQ 0.18.0, everything is empty

Attributes: 
sub=>..., 
nbf=>Wed Sep 22 08:26:30 UTC 2021, 
connectsFilterRegexp=>[], 
roles=>[], 
iss=>akhq, 
exp=>Wed Sep 22 09:26:30 UTC 2021, 
iat=>Wed Sep 22 08:26:30 UTC 2021, 
consumerGroupsFilterRegexp=>[], 
topicsFilterRegexp=>[]

I saw a lot of changes around OIDC in AKHQ 0.17 and 0.18. and somewhere there the configuration is lost. Do you have an idea why our setup isn't working anymore or what is missing in our JWT Token which is required by AKHQ?

[wilkej]

I did further testing and I could get it working if I add a group defintion to my application.yaml and define this group name as role in the Keycloak Client.

I understood that AKHQ checks if the provided role is found as group definition in the configuration file. This is very unlucky for me.

Currently in 0.16. I can manage all permissions via Keycloak and add or remove permissions on the fly. With the new changes in 0.17 and 0.18 and need to define all groups in the AKHQ config and need to restart AKHQ to do permission changes. Currently I have around 30 different groups with different permissions per stage. It would be a big effort to migrate all configurations to the AKHQ config.

Is there any option to bypass the check for the group configuration in the application.yaml?

[Dujma]

I get the same behavior with Azure AD. Not sure if it's the same problem as it is in Keycloak, but I do get the empty JWT in the cookie.

[tchiotludo]

@twobeeb : is this can be a side effect of https://github.com/tchiotludo/akhq/pull/678, you think ?

[Dujma]

Is there an easy fix or a way to bypass this at the moment?

[tchiotludo]

@Dujma don't know ! The whole implementation is done by external people and I don't have any Keycloak to test. My only usage is based on Google signon and so I need to put myself the group and the role in akhq since Google don't provide this.

[Dujma]

Thanks for the replies! I will play with this a bit today and if I manage to find a solution, I'll share it here :)

[wilkej]

I think it isn't a direct side effect of #678 as the issue also occurs in 0.17.0 Based what I researched - be warned I'm not a Java Developer ;-) - I think our issue started here: https://github.com/tchiotludo/akhq/pull/573

The following code was added

        /**
        * In case of OIDC the user roles are not correctly mapped to corresponding roles in akhq,
        * If we find a groups-field in the user attributes override it with the correctly mapped
        * roles that match the associated akhq group
        */
        Oidc.Provider provider = oidc.getProvider(providerName);
        if (attributes.containsKey(provider.getGroupsField())) {
            attributes.put(provider.getGroupsField(), roles);
        }

As in our application.yaml is no group definition, this will not match and therefore the token is empty. Following #678 I couldn't identify it exactly but assume the reason somewhere near src/main/java/org/akhq/utils/LocalSecurityClaimProvider.java In the mapToAkhqGroups function.

[wilkej]

Maybe this is a solution, if OIDC / JWT is used, check if the token include role definitions which map with AKHQ role definitions e.g. topic/read. If they match, take them directly as roles into the AKHQ token and do not map the token roles to AKHQ groups. The same behaviour should apply for the topic, connect and consumer group regexp. From my point of view the AKHQ groups (default as self defined in the application.yaml) could be ignored for OIDC/JWT.

[twobeeb]

@wilkej you're correct with your analysis. As of 0.18 I refactored the OIDC code without changing it (or so I hope)

My understanding when I refactored is that OIDC provider should provide AKHQ with "OIDC" groups, and then those groups will translate to "AKHQ" groups which finally generates the final roles/attributes list. In my opinion it was already the intented way in 0.16 : https://github.com/tchiotludo/akhq/blob/0.16.0/src/main/java/org/akhq/modules/OidcUserDetailsMapper.java

• Takes groups from OIDC
• Translate to AKHQ groups
• Map to roles/attributes

In your case keycloak already provides AKHQ with roles and attributes "ready to go". That's not supported with current code.

If @tchiotludo is fine with such design, I think it's easy to add a parameter oidc.providers.keycloak.use-oidc-claim: true to by pass the whole logic currently in place and generate the JWT without going through AKHQ groups.

Small change would be needed in https://github.com/tchiotludo/akhq/blob/dev/src/main/java/org/akhq/modules/OidcUserDetailsMapper.java#L52 to check whether we need to go with ClaimProvider model or rather stay in OIDC direct.

Anyway, PR welcome

[Dujma]

I've resolved this by matching the groups from Azure AD to groups from AKHQ and by setting the groups-field: groups in the application.yml.

It seems that my issue was a bit different than the one originally reported.

[wilkej]

@Dujma could you post your application configuration?

You wrote you match the Azure AD groups to AKHQ. Does this mean you use the AKHQ default groups?

[Dujma]

@wilkej here you go :)

micronaut:
  server:
    host-resolution:
      protocol-header: "X-Forwarded-Proto"
      host-header: "host"
      port-header: "X-Forwarded-Port"
      port-in-host: false
    port: <port>
  security:
    authentication: cookie
    enabled: true
    endpoints:
      oauth:
        enabled: true
    session:
      enabled: false
    token:
      jwt:
        bearer:
          enabled: false
        enabled: true
        cookie:
          enabled: true
          cookie-same-site: 'Lax'
          cookie-secure: true
        signatures:
          secret:
            generator:
              secret: "<jwt_secret>"
              base64: true
              jws-algorithm: HS256
    oauth2:
      enabled: true
      clients:
        oidc:
          client-id: "<oidc_client_id>"
          client-secret: "<oidc_client_secret>"
          openid:
            issuer: "<issuer>"

akhq:
  ui-options:
    topic-data:
      sort: Newest
    topic:
      skip-last-record: true

  security:
    default-group: no-roles
    groups:
      <group-id-from-azure-ad>:
        name: <group-name>
        roles:
          - topic/read
          - topic/insert
          - topic/delete
          - topic/config/update
          - node/read
          - node/config/update
          - topic/data/read
          - topic/data/insert
          - topic/data/delete
          - group/read
          - group/delete
          - group/offsets/update
          - registry/read
          - registry/insert
          - registry/update
          - registry/delete
          - registry/version/delete
          - acls/read
          - connect/read
          - connect/insert
          - connect/update
          - connect/delete
          - connect/state/update
      <other-group-id-from-azure-ad>:
        name: <other-group-name>
        roles:
          - topic/delete
          - topic/config/update
          - topic/data/read
          - topic/data/insert
          - topic/data/delete
          - connect/read
          - connect/insert
          - connect/update
          - connect/delete
          - connect/state/update
          - group/offsets/update
    oidc:
      enabled: true
      providers:
        oidc:
          label: "<label>"
          username-field: unique_name
          groups-field: groups
          groups:
            - name: <group-id-from-azure-ad>
              groups:
                - <group-name>
            - name: <other-group-id-from-azure-ad>
              groups:
                - <other-group-name>

  server:
    access-log:
      enabled: false
      name: org.akhq.log.access

  connections:
    data-pipeline:
      properties:
        bootstrap.servers: <kafka-brokers>

[wilkej]

Thank you. I understand you define the groups in the application.yaml - I guess this is how the development team expected that it works. If you want to add a group or change permissions for a group you need to restart the application to load the new application configuration. With our 0.16 setup we were able to manage the permission complete dynamically without changing the application configuration

[twobeeb]

@wilkej Please note that other than my PR suggestion above, you can use external authentication mechanism with API call :

akhq:
  security:
    rest:
      enabled: true
      url: http://external.auth.service.com/claim-handler

Exposed API must accept a POST with input payload AKHQClaimRequest and reply a AKHQClaimResponse https://github.com/tchiotludo/akhq/blob/dev/src/main/java/org/akhq/utils/ClaimProvider.java

This requires some minor development on your side but it will also give you more freedom. Exemple implementation : https://github.com/michelin/ns4kafka/blob/master/api/src/main/java/com/michelin/ns4kafka/controllers/AkhqClaimProviderController.java

[wilkej]

@twobeeb Thank you for the hint. I don't know how to put it but for me this isn't really a solution. I would need something which provides this endpoint and this something must sync with our keycloak.

As you wrote "In your case keycloak already provides AKHQ with roles and attributes "ready to go". That's not supported with current code."

For me it is a loss of functionality but I'm in the unfortunate situation that I can't provide a PR for this :(

[twobeeb]

I understand.

I may find some time to implement oidc.providers.<service>.use-oidc-claim: true to suit your use case (and bring back your functionality loss) in the next few weeks if nobody takes it until then.

[wilkej]

Thank you. This is highly appreciated.

[tchiotludo]

@wilkej even if you are not a java developer, maybe you can add a unit test (that will failed) with a token you have here, this part is completely blindness for me, have a good token as a user can have will help to add a unit test about that

[wilkej]

@tchiotludo I extend the docker compose file to include a pre-configured keycloak instance to creates different tokens for different users.

I'd like to contribute this and I wonder where to put it. I would also add some readme specific for keycloak for people who didn't have experience with it. My suggestion is to create an example folder with sub-folder keycloak where I put the docker compose file and the readme.

This is an token created by the local keycloak instance.

{
  "exp": 1635756149,
  "iat": 1635755849,
  "jti": "95a452f2-7e97-4982-ae53-a320058e2015",
  "iss": "http://localhost:8666/auth/realms/akhq",
  "sub": "636969cb-13a1-4533-9e55-d148cee9682d",
  "typ": "Bearer",
  "azp": "akhq-test",
  "session_state": "8ecace77-d73b-42ef-b4db-87eacc724ae5",
  "acr": "1",
  "allowed-origins": [],
  "resource_access": {
    "akhq-test": {
      "roles": [
        "topic/data/delete",
        "acls/read",
        "topic/data/insert",
        "topic/read",
        "group/read",
        "node/read",
        "topic/data/read"
      ]
    }
  },
  "scope": "openid email profile",
  "sid": "8ecace77-d73b-42ef-b4db-87eacc724ae5",
  "topics-filter-regexp": [
    "^csv.*$"
  ],
  "email_verified": false,
  "roles": [
    "topic/data/delete",
    "acls/read",
    "topic/data/insert",
    "topic/read",
    "group/read",
    "node/read",
    "topic/data/read"
  ],
  "name": "csv reader",
  "preferred_username": "csv",
  "given_name": "csv",
  "family_name": "reader",
  "email": "csv@akhq.org"
}

[tchiotludo]

@wilkej thanks for sharing that ! Now AKHQ have a full documentation website here with source here maybe you can add to this page the full example or a subpage ?

[wilkej]

I tried to add some documentation and came to the conclusion that this would be really focussed on keycloak. As AKHQ is focused on Kafka I think it will lead in the wrong direction.

I created an additional token which includes a consumer-groups-filter-regexp. With this I hope it should be possible to implement the skip option @twobeeb mentioned.

{
  "exp": 1635868816,
  "iat": 1635868516,
  "jti": "2e8ed4f1-7129-4986-943e-ae85e6df3a15",
  "iss": "http://localhost:8666/auth/realms/akhq",
  "sub": "524cd1d2-01cf-441e-a8b8-04f872e69aee",
  "typ": "Bearer",
  "azp": "akhq-test",
  "session_state": "425dcdb8-9743-464e-b8ce-6b4f4723df30",
  "acr": "1",
  "allowed-origins": [],
  "resource_access": {
    "akhq-test": {
      "roles": [
        "acls/read",
        "topic/data/delete",
        "topic/data/insert",
        "topic/read",
        "group/read",
        "node/read",
        "topic/data/read"
      ]
    }
  },
  "scope": "openid email profile",
  "sid": "425dcdb8-9743-464e-b8ce-6b4f4723df30",
  "topics-filter-regexp": [
    "^json.*$"
  ],
  "email_verified": false,
  "roles": [
    "acls/read",
    "topic/data/delete",
    "topic/data/insert",
    "topic/read",
    "group/read",
    "node/read",
    "topic/data/read"
  ],
  "name": "json reader",
  "consumer-groups-filter-regexp": [
    "^json-consumer.*$"
  ],
  "preferred_username": "json",
  "given_name": "json",
  "family_name": "reader",
  "email": "json@akhq.org"
}

[tchiotludo]

@wilkej so in conclusion ?

• roles in the oidc claims is not used in akhq any more ?
• you need to define groups in akhq configuration ?
• if true, you also need to defined the user in akhq ?

[wilkej]

That isn't what I meant to say :)

I still like the functionality of AKHQ 0.16 back where my roles of the oidc claim will be used. I don't want to define the groups and users in the AKHQ config.

I meant I don't want to overload the documentation with our keycloak configuration. I'm worried it could cause additional questions or issues. Therefore leave the documentation at it is.

[twobeeb]

@wilkej Check this ? #933

[wilkej]

Thank you. I'll try this in this week.

[wilkej]

Hello @twobeeb ,

I was able to test it with your feature branch and it looks good.

As a side note I saw that the regex filter object name changed. In the 0.16 version we have a claim called topics-filter-regexp and now it is called topicsFilterRegexp. After changing it it works. Did this changes with your changes in 0.17?

I would be happy if your changes will be available in an other release.

[twobeeb]

@wilkej I'm not sure. I made a change last year to add connect filter and it was still using kebab-case : https://github.com/tchiotludo/akhq/pull/477/files#diff-3b2f1e772c4c6487ff3cb9cf2a75ee2d0f5a28e0685e7d715e9897ee9507477cR259

This MR changes it to camelCase : https://github.com/tchiotludo/akhq/pull/578

Anyway, if you're fine with changing the claim on your side to camelCase, I will add on more test today or tomorow and merge it.

[wilkej]

@twobeeb I'm fine changing to camelCase, thank you :)

In the documentation https://akhq.io/docs/configuration/authentifications/groups.html the filter are written in kebab-case. Out of curiosity where is the magic happening to changes this to camelCase

[twobeeb]

Micronaut framework does the magic : https://docs.micronaut.io/latest/guide/#immutableConfig

Configuration injection resolves kebab-case from yaml configuration files into their java camel case equivalent variables if you have a dedicated class for it. Which there is : https://github.com/tchiotludo/akhq/blob/dev/src/main/java/org/akhq/configs/SecurityProperties.java Big subject. There's a lot more to it, like seamless support for env variables (uppercase with underscores).