vthorwat commented 4 years ago

Hello ,

We are planning to implement LDAP Authentication to Kafka Hq.

We are trying with below configurations but no luck with it.

Application.yml

micronaut: security: ldap: default: enabled: true context: server: 'ldaps://adldap.abc.xyz.com:3269' managerDn: 'CN=BIND_USER,OU=SERVICE ACCT,DC=abc,DC=xyz,DC=com' managerPassword: 'D$ve&9IL' search: base: "DC=xyz,DC=com" groups: enabled: true base: "DC=xyz,DC=com" server: port: 7070

kafkahq: connections: jbdl-dev: properties: bootstrap.servers: "BROKER:6667,BROKER:6667,BROKER:6667" security.protocol: SASL_PLAINTEXT authorizer.class.name: kafka.security.auth.SimpleAclAuthorizer sasl.kerberos.service.name: kafka sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" principal="kafka/HOSTNAME@REALM.COM";

security: default-groups:

admin

groups: admin: roles:

topic/read
topic/insert
topic/delete
topic/config/update
node/read
node/config/update
topic/data/read
topic/data/insert
topic/data/delete
group/read
group/delete
group/offsets/update
registry/read
registry/insert
registry/update
registry/delete
registry/version/delete
acls/read
connect/read
connect/insert
connect/update
connect/delete
connect/state/update

topic-reader: roles:

topic/read

endpoints: env: enabled: true sensitive: false

====================================================== If anyone has implemented LDAP Authentication to KafkaHq their help is appreciated.

Thanks in advance.

tchiotludo commented 4 years ago

If you want to have help, please format your yaml to be readable ... Nobody can help you without format.

I don't see any mapping between ldap group and kafkahq group in your yaml :

kafkahq:
  security:
    ldap:
      group:
        mathematicians: # LDAP Group 
          groups:
            - topic-reader # Kafka HQ groups

You can also debug ldap with this curl :

curl -i -X POST -H "Content-Type: application/json" \
       -d '{ "configuredLevel": "TRACE" }' \
       http://localhost:8080/loggers/io.micronaut.configuration.security

vthorwat commented 4 years ago

Hi ,

After Mapping of my Admin LDAP group to admin group in KAfkaHq still I am not able to login to the page .

ERROR LOG :

login_error_kafkahq

Please find the Group mapping in app.yml:

security: default-groups:

admin

groups: admin: roles:

topic/read
topic/insert
topic/delete
topic/config/update
node/read
node/config/update
topic/data/read
topic/data/insert
topic/data/delete
group/read
group/delete
group/offsets/update
registry/read
registry/insert
registry/update
registry/delete
registry/version/delete
acls/read
connect/read
connect/insert
connect/update
connect/delete
connect/state/update

topic-reader: roles:

topic/read

ldap: group: GROUP--TOOLS--AMBARIMGMT: groups:

admin

=================================================

LDAP CONFIG

micronaut: security: ldap: default: enabled: true context: server: 'ldaps://adldap.xyz.abc.com:3269' managerDn: 'CN=BIND_USER,OU=SERVICE ACCT,DC=xyz,DC=abc,DC=com' managerPassword: 'BINDUSER_PASSWORD' search: base: "DC=abc,DC=com" groups: enabled: true base: "OU=BDL_DEV,OU=GROUPS,OU=IT,OU=MEMBER SERVERS,DC=xyz,DC=abc,DC=com"

===============================================

One more thing when i try to open Login page for KafkaHq it is showing us UNAUTHORIZED in URL DESCRIPTION as below but it still shows the login page.

Login_Page_issue

vthorwat commented 4 years ago

One question I am having regarding LDAP mapping .

We have to configure it wihin "kafkahq:" or in "micronaut:"

I am currently configuring it in the "kafkahq:".

tchiotludo commented 4 years ago

To properly format code, use https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code please, still unreadable !

Mapping must be done in kafkahq: , micronaut: configuration is necessary only for ldap server configuration.

For the the mapping to work with special characters, just must use the trick here https://github.com/tchiotludo/kafkahq/issues/178. so must be a configuration (part) like :

security:
  ldap:
    group:
      ambarimgmt: # not really important id that will be override by name below
        name: GROUP--TOOLS--AMBARIMGMT
        groups:
         - admin

vthorwat commented 4 years ago

Thanks for reply..

I tried with the given changes but still USER NOT FOUND error is getting while login to KafkaHq

Please find the application.yml :

micronaut:
  security:
     ldap:
       default:
         enabled: true
         context:
              server: 'ldaps://adldap.xyz.abc.com:3269'
              managerDn: 'CN=BIND_USER,OU=SERVICE ACCT,DC=xyz,DC=abc,DC=com'
              managerPassword: 'BIND_PASSWORD'
         search:
              base: "DC=abc,DC=com"
         groups:
              enabled: true
              base: "OU=BDL_DEV,OU=GROUPS,OU=IT,OU=MEMBER SERVERS,DC=xyz,DC=abc,DC=com"

  server:
    port: 7070
kafkahq:
 connections:
    jbdl-dev:
      properties:
        bootstrap.servers: "BROKER:6667,BROKER:6667,BROKER:6667"
        security.protocol: SASL_PLAINTEXT
        authorizer.class.name: kafka.security.auth.SimpleAclAuthorizer
        sasl.kerberos.service.name: kafka
        sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" principal="kafka/HOSTNAME@REALM.COM";

 security:
   default-groups:
        -
 groups:
   admin:
    roles:
      - topic/read
      - topic/insert
      - topic/delete
      - topic/config/update
      - node/read
      - node/config/update
      - topic/data/read
      - topic/data/insert
      - topic/data/delete
      - group/read
      - group/delete
      - group/offsets/update
      - registry/read
      - registry/insert
      - registry/update
      - registry/delete
      - registry/version/delete
      - acls/read
      - connect/read
      - connect/insert
      - connect/update
      - connect/delete
      - connect/state/update
    attributes:
     topics-filter-regexp: "test.*"

   topic-reader:
     roles:
       - topic/read
     attributes:
       topics-filter-regexp: "test\\.reader.*"
 ldap:
   group:
     ambari:
       name: GROUP--TOOLS--AMBARIMGMT
       groups:
         - admin
         - topic-reader
endpoints:
    env:
    enabled: true
    sensitive: false

Please let me know if any other changes required.

tchiotludo commented 4 years ago

Each ldap is a different kind of monster ... Really hard to debug remotely.

All the Ldap configuration is micronaut based here : https://micronaut-projects.github.io/micronaut-security/latest/guide/#ldap

Most of the configuration is simple mapping from java. Don't you have a java application connected to ldap in your org to help you ?

vthorwat commented 4 years ago

Yes we are validating the same. But looks all ok.

Still figuring why this is occuring.

As from the configurations we are able to understand that configurations for connecting to ldap are connect only problem is We are not able to search the user in AD.

We are filtering the groups in config like this :

micronaut:
  security:
      ldap:
        default:
          enabled: true
          context:
               server: 'ldaps://adldap.xyz.abc.com:3269'
               managerDn: 'cn=BIND_USER,ou=SERVICE ACCT,dc=xyz,dc=abc,dc=com'
               managerPassword: 'PASSWORD'
          search:
               enabled: true
               base: "dc=ril,dc=com"
          groups:
               enabled: true
               base: "ou=BDL_DEV,ou=GROUPS,ou=IT,ou=MEMBER SERVERS,dc=xyz,dc=abc,dc=com"
               filter: "cn=GALL--DEVBDL--*"
               attribute: "cn"

Please can you just check whether is correct or not?

vthorwat commented 4 years ago

Hello,

We have tried all the necessary steps and validated configs for app.yml, but still not working.

We were not able to Search user in Base DN. The same base DN is configured for other services in our environment & still it works.. but not with KafkaHq.

If anyone have any idea regarding ldap configurations please assist for the same.

I want to thank you @tchiotludo for you kind support.

Keep this open will update as if we find out the issue..

vthorwat commented 4 years ago

DEBUG :

2020-01-24 18:06:39,027 DEBUG pGroup-1-9 LdapAuthenticationProvider Starting authentication with configuration [default]
2020-01-24 18:06:39,027 DEBUG pGroup-1-9 LdapAuthenticationProvider Attempting to initialize manager context
2020-01-24 18:06:39,398 DEBUG pGroup-1-9 LdapAuthenticationProvider Manager context initialized successfully
2020-01-24 18:06:39,398 DEBUG pGroup-1-9 LdapAuthenticationProvider Attempting to authenticate with user [User.Name]
2020-01-24 18:06:39,419 DEBUG pGroup-1-9 LdapAuthenticationProvider User not found [User.Name]
2020-01-24 18:06:39,424 INFO pGroup-1-9 org.kafkahq.log.access [Date: 2020-01-24T18:06:38.913722+05:30] [Duration: 510 ms] [Url: POST /login HTTP/1.1] [Status: 303] [Ip: 10.77.245.201] [Length: 0] [Port: 7070]
2020-01-24 18:06:39,444 INFO Group-1-10 org.kafkahq.log.access [Date: 2020-01-24T18:06:39.439105+05:30] [Duration: 4 ms] [Url: GET /login/failed HTTP/1.1] [Status: 200] [Ip: 10.77.245.201] [Length: 679] [Port: 7070]

This what we get now.

vthorwat commented 4 years ago

Which user will be accessing in background while using KafkaHq.jar ?

As in the configurations I am specifying Kafka user's keytab and principal to run & as well to run Jar from command line we are doing this by Kafka user only.

But Strange behavior we are facing while accessing the TOPIC from GUI.

Error :

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [Topic authorization failed.]

As we are having full access policy for Kafka user in our Apache Ranger Environment.

If we could get the proper user which is running through JAR will add the policy for the same.

For this step still we are having blocker at LDAP integration .

tchiotludo commented 4 years ago

The user connecting Kafka cluster is the one that you provide on the connections part of the configuration. If you want to have access, you need to provide a valid user from Kafka acl. Like I said, I never have a Kerberos auth cluster so I'm not aware of the configuration, sorry ☹️

HungUnicorn commented 4 years ago

@vthorwat have you tried the ldap online test server and discover sth? https://www.forumsys.com/tutorials/integration-how-to/ldap/online-ldap-test-server/

I tested the login with the ldap online test server and a local kafkahq, and it works for me. It would be simpler to debug with a common ldap server so the issue can be reproduced locally

tchiotludo commented 4 years ago

Closing due to inactivity, please reopen if needed

tchiotludo / akhq

LDAP Configuration for KafkaHQ #197

Application.yml