LDAP Config and questions

jessetandor commented 4 years ago

Alil difficulty with LDAP on our end - here is our config :

micronaut:
  security:
    # Ldap authentificaton configuration
    ldap:
      default:
        enabled: true
        context:
          server: 'ldap://local.ad.company.net:xxx' 
          managerDn: 'cn=user,ou=xxx,ou=Company Groups,dc=ad,dc=company,dc=net'
          managerPassword: '123'
        search:
          base: "dc=ad,DC=company,dc=net"
        groups:
          enabled: true
          base: "dc=ad,DC=company,dc=net"  

kafkahq:
  connections:
    Dev-Region-Company-Cluster:
      properties:
        bootstrap.servers: "x.x.x.x:xxx" # IP and Port of one machine in the cluster for that region
        security.protocol: SASL_PLAINTEXT # This is our setup in Kafka
        sasl.mechanism: SCRAM-SHA-256 # This is our setup in Kafka
        sasl.jaas.config: "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"xxx\";" # This has to match the user creaked in Kafka

  pagination:   
    page-size: 5 # number of elements per page 
    threads: 20 # Number of parallel threads to resolve page - can be performance heavy

  security:
    # Groups definition
    groups:
      admin: # Group name in kafkaHQ
        roles:  # roles for the group
          - topic/read
          - topic/insert
          - topic/delete
          - topic/config/update
          - node/read
          - node/config/update
          - topic/data/read
          - topic/data/insert
          - topic/data/delete
          - group/read
          - group/delete
          - group/offsets/update
          - registry/read
          - registry/insert
          - registry/update
          - registry/delete
          - registry/version/delete
          - acls/read
          - connect/read
          - connect/insert
          - connect/update
          - connect/delete
          - connect/state/update
      readonly: 
        roles:
          - topic/read
          - node/read
          - topic/data/read
          - group/read
          - registry/read
          - connect/read

    # Ldap Groups configuration (when using ldap)
    ldap:
      group:
        kafkahq-admin: # ldap group name
          groups: # KafkaHQ groups list
            - admin
        kafkahq-readonly:
          groups:
            - readonly

Trace has been enabled in the output and we noticed the LDAP server was being contacted but on port 9092..

1) Is it expected that the kafka cluster is also setup using LDAP? 2) If not - how does KafkaHQ request a login screen? 3) Are there any other way's to debug LDAP side? 4) Any plans to support LDAPS?

tchiotludo commented 4 years ago

Hi,

First of all, do you use the dev version ? the release is not done yet for now (coming soon).

For the question :

No LDAP auth from kafkaHQ is completely independent from the Kafka One, no need to configure the Kafka cluster auth.
KafkaHQ will enable a login page if the configuration is right and detect the any auth (ldap or basic).
To be honest, I don't know, I never try but you can try to raise the log level on io.micronaut.configuration.security.ldap to have some insight.
Ldaps seems to be supported by micronaut directly as I see (never try).

Here is a minimal configuration with ldap & basic auth :

micronaut:
  security:
    enabled: true
    ldap:
      default:
        enabled: true
        context:
          server: 'ldap://ldap.forumsys.com:389'
          managerDn: 'cn=read-only-admin,dc=example,dc=com'
          managerPassword: 'password'
        search:
          base: "dc=example,dc=com"
        groups:
          enabled: true
          base: "dc=example,dc=com"

kafkahq:
  server:
  connections:
    plaintext:
      properties:
        bootstrap.servers: "kafka:9092"
      schema-registry:
        url: "http://schema-registry:8085"
        basic-auth-username: test_user
        basic-auth-password: test_pass
      connect:
        url: "http://connect:8083"
        basic-auth-username: test_user
        basic-auth-password: test_pass
      ksql:
        url: "http://ksql:8088"

  security:
    basic-auth:
      user:
        password: d74ff0ee8da3b9806b18c877dbf29bbde50b5bd8e4dad7a3a725000feb82e8f1
        groups:
        - admin

    ldap:
      group:
        mathematicians:
          groups:
            - reader
        scientists:
          groups:
            - admin

user / pass for basicauth user & pass from here for ldap

jessetandor commented 4 years ago

Yap - running Dev version from Docker. Making progress but run into a small issue with AD Cert

LdapAuthenticationProvider Failed to create manager context. Returning unknown authentication failure. Encountered {}
`javax.naming.CommunicationException: simple bind failed: local.xx.xx.xx:xx`

I'd like to test without introducing the certificate. How can we completely disable or ignore certificate validation?

I tried :

$ curl -i -X POST -H "Content-Type: application/json" \
>        -d '{ "configuredLevel": “FALSE” }' \
>        http://localhost:5000/Dcom.sun.net.ssl.checkRevocation
HTTP/1.1 401 Unauthorized
Date: Mon, 9 Dec 2019 13:49:08 GMT
transfer-encoding: chunked
connection: close

But the result is 401. Most likely this method is not supported anymore as it's very old (https://tinyurl.com/y4hvdgfp)

Adding relevant sections below:

micronaut:
  security:
    # Ldap authentificaton configuration
    ldap:
      default:
        enabled: true
        context:
          server: 'ldaps://xxx.xx.xxx.xxx:xxx' 
          managerDn: 'cn=xxx,dc=xx,dc=xx,dc=xx'  
          managerPassword: '123'
        search:
          base: "dc=company,dc=net"
        groups:
          enabled: true
          base: "dc=company,dc=net"

    # Ldap Groups configuration (when using ldap)
    ldap:
      group:
        kafkahq-admin: # ldap group name
          groups: # KafkaHQ groups list
            - admin
        kafkahq-readonly:
          groups:
            - readonly

tchiotludo commented 4 years ago

To be honest, I don't know anything about ldaps, never used myself.

In general, in java, the certificate is mandatory in java, and you will need a custom keystore with your certificate in it : http://roufid.com/java-ldap-ssl-authentication/#1-_register_your_ssl_certificate_into_java_keystore.

I don't know if the -Dcom.sun.net.ssl.checkRevocation=false is still valid but it's a java opts.

You need to pass it to docker image with env vars, example here for docker-compose

    environment:
      JAVA_OPTS: "-Dcom.sun.net.ssl.checkRevocation=false"

jessetandor commented 4 years ago

Yap, that helped tchiotludo. I was able to catch logs and eventually no Cert needed to be set for the managerDN login. Your DN has to be 100% correct ofc, mine was slightly off.

I have run into another issue though, maybe you know but AD power users might know more.

Ive created two Groups

distinguishedName: CN=kafkahq-readonly,OU=Kafka,OU=company Groups,DC=ad,DC=company,DC=net
distinguishedName: CN=kafkahq-admin,OU=Kafka,OU=company Groups,DC=ad,DC=company,DC=net

And called them in my LDAP section

    # Ldap Groups configuration (when using ldap)
    ldap:
      group:
        kafkahq-admin: # ldap group name
          groups: # KafkaHQ groups list
            - admin
        kafkahq-readonly:
          groups:
            - readonly

However when trying to login ( a login screen is presented so the connection is there ) - i get

2019-12-11 11:47:19,273 DEBUG pGroup-1-2 LdapAuthenticationProvider Starting authentication with configuration [default]
2019-12-11 11:47:19,273 DEBUG pGroup-1-2 LdapAuthenticationProvider Attempting to initialize manager context
2019-12-11 11:47:20,372 DEBUG pGroup-1-2 LdapAuthenticationProvider Manager context initialized successfully
2019-12-11 11:47:20,373 DEBUG pGroup-1-2 LdapAuthenticationProvider Attempting to authenticate with user [user]
2019-12-11 11:47:20,537 DEBUG pGroup-1-2 LdapAuthenticationProvider User found in context [CN=user,OU=company Users,DC=ad,DC=company,DC=net]. Attempting to bind.
2019-12-11 11:47:21,041 DEBUG pGroup-1-2 LdapAuthenticationProvider Successfully bound user [CN=user,OU=company Users,DC=ad,DC=company,DC=net]. Attempting to retrieving groups.
2019-12-11 11:47:21,183 DEBUG pGroup-1-2 .DefaultLdapGroupProcessor No groups found!
2019-12-11 11:47:21,184 DEBUG pGroup-1-2 .DefaultLdapGroupProcessor Attempting to add any additional groups...
2019-12-11 11:47:21,184 DEBUG pGroup-1-2 LdapAuthenticationProvider Group search returned [[]] for user [user]
2019-12-11 11:47:21,185 TRACE pGroup-1-2 LdapAuthenticationProvider Attempting to map [user] with groups [[]] to an authentication response.
2019-12-11 11:47:21,188 DEBUG pGroup-1-2 LdapAuthenticationProvider Response successfully created for [user]. Response is authenticated: [true]

So the manager login is succesful, the user credentials are also correct as the user is bound, but no groups are found for the user thus i get Forbidden on the web side.

I cannot figure our why the groups aren't detected : i cant see wrong with my settings either:

micronaut:
  security:
    # Ldap authentificaton configuration
    ldap:
      default:
        enabled: true
        context:
          server: 'ldaps://local.ad.company.net:636' 
          managerDn: 'CN=users,OU=xxx,OU=xxxx,DC=xx,DC=xxx,DC=xxx'  
          managerPassword: 'password_1234'
        search:
          base: "OU=xxx,DC=xx,DC=xx,DC=xx"
        groups:
          enabled: true
          base: "OU=xxx,DC=xx,DC=xx,DC=xx"

tchiotludo commented 4 years ago

As I see on code, the group base configuration don't return any things. So it's not you kafkahq group that failed, but it was the ldap group that is not found if it can help .

Do you have try with test ldap server first ?

micronaut:
  security:
    enabled: true
    ldap:
      default:
        enabled: true
        context:
          server: 'ldap://ldap.forumsys.com:389'
          managerDn: 'cn=read-only-admin,dc=example,dc=com'
          managerPassword: 'password'
        search:
          base: "dc=example,dc=com"
        groups:
          enabled: true
          base: "dc=example,dc=com"

to make it work and adapt it for your local company ad ?

jessetandor commented 4 years ago

Yes, the two LDAP groups arent' found, even though my user is a member of the admin group.

Based on your example if i change the search base to just : "DC=ad,DC=company,DC=net", it fails to authenticate the user logging in.

'2019-12-11 13:46:04,553 DEBUG pGroup-1-2 LdapAuthenticationProvider Failed to authenticate with user  {}
javax.naming.PartialResultException: Unprocessed Continuation Reference(s)

And yes, i've been testing with multiple AD setup's 👍

tchiotludo commented 4 years ago

The configuration seems to be good. Seems that your user is not in any groups on AD. Are you sure that you have any group with user tandor ?

The log with no groups is coming from micronaut before mapping from KafkaHQ, so must be the group lookup that failed with groups.base configuration.

The only thing I see is that user and group is not on the same hierarchy but unfortunately, I'm not a boss in AD at all so hard to help. Sorry ...

jessetandor commented 4 years ago

Yes, my user is definitely part of the kafka group in AD

Are there any extra arguments we can pass in the group's LDAP section? Like filter or matchGroup ? We've had to do this before with Cerebro and that seemed to work.

tchiotludo commented 4 years ago

all the option are here : https://micronaut-projects.github.io/micronaut-security/latest/guide/#ldap

There is option like filter for your use case, maybe on this will be working ?

jessetandor commented 4 years ago

Yep, so we gave up on this - it doesnt seem to search nested values, thus no groups are found. We also tried to hack it alil from micronaut's side : 1) Using Member/Memberof attributes 2) Direct Filters to narrow down the group detection (directly calling CN values)

Sadly these still detected no groups from Samba AD server.

Going back to Read Only mode: @tchiotludo Did you remove it from version 0.11.0 and up?

My config file works in read-only mode for 0.10.0 but in 0.11.0, i can see the button for topic creation...again it's the same file and syntax is correct.

kafkahq:
  connections:
    test:
      properties:
        bootstrap.servers: "x.x.x.x:9092" # IP and Port of one machine in the cluster for that region
        security.protocol: SASL_PLAINTEXT # This is our setup in Kafka
        sasl.mechanism: SCRAM-SHA-256 # This is our setup in Kafka
        sasl.jaas.config: "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"xxxx\";" # This has to match the user creaked in Kafka

  pagination:   
    page-size: 10 # number of elements per page 
    threads: 64 # Number of parallel threads to resolve page - can be performance heavy

  security:
    default-roles:
      - topic/read
      - node/read
      - topic/data/read
      - group/read
      - registry/read
      - connect/read

tchiotludo commented 4 years ago

The configuration have change : https://github.com/tchiotludo/kafkahq/blob/a6ebf24f8c0ac2936397361aaf4ff4d3d4f706c4/application.example.yml#L105

Here is an example to have a reader by default and 1 user with all acls :

  security:
    default-groups: # Default groups for all the user even unlogged user
      - reader
    basic-auth:
      user:
        password: d74ff0ee8da3b9806b18c877dbf29bbde50b5bd8e4dad7a3a725000feb82e8f1
        groups:
        - admin

jessetandor commented 4 years ago

We are up and running with 0.11.0. Thanks alot..

You mentioned Multi cluster in the features - what do you mean by this? As far as I've read, one instance of kafkaHQ can only connect to one or many machine's in the same kafka cluster. Or is that wrong?

About the ko - mo - go values for weights, what do they mean - log dir size?

tchiotludo commented 4 years ago

Glad to know it's working, but some kind of frustrated that you can't have the ldap running.

Multi cluster mean that you can connect to multiple Kafka cluster on the same KafkaHQ instance (like dev, prod, ...). Just duplicate the connections on the configuration files to have multiple cluster.

Weight is log dir size, maybe should rename the cold 😂

Closing this one, since it's more discussion !

Feel free to add a PR on #161 😁

skaravad commented 3 years ago

@jessetandor @tchiotludo may be you have already got this working, I ran into similar issue and I was able to fix it. First of all, it all depends on the type of LDAP you use, the default micronaut config for group search filter is set to "uniquemember={0}" which might not hold true for Active Directory , from my vague memory it could be member (so the setting will be "member ={0}"

Here is the config section under micronaut

micronaut:
  security:
    ldap:
      default:
        enabled: true
        context:
          server: 'ldaps://ldapserver:636'
          managerDn: 'CN=xxx,CN=Users,DC=MyDomain,DC=com"
          managerPassword: 'xxxxxxxx'
        search:
          base: "CN=Users,DC=MyDomain,DC=com"
        groups:
          enabled: true
          subtree: true  # Pay attention here, this is needed.
          base: "CN=Users,DC=MyDomain,DC=com"
          filter: 'member={0}'`

Under the akhq section, 
 security:
    default-group: reader # Default groups for all the user even unlogged user
    # Groups definition
    groups:
      reader:
        roles: ""
      admin: # Group name
        roles:  # roles for the group
          - topic/read
          - topic/insert
          - topic/delete
          - topic/config/update
          - node/read
          - node/config/update
          - topic/data/read
          - topic/data/insert
          - topic/data/delete
          - group/read
          - group/delete
          - group/offsets/update
          - registry/read
          - registry/insert
          - registry/update
          - registry/delete
          - registry/version/delete
          - acls/read
          - connect/read
          - connect/insert
          - connect/update
          - connect/delete
          - connect/state/update
    ldap:
      group:
        ldapAkHq-admins:   # This is your active directory group name just the part of the CN
          groups:
            - admin
    basic-auth:

tchiotludo commented 3 years ago

thanks @skaravad for help :)

gerardgorrion commented 3 years ago

Hi skaravad,

I was tryed to configurate my LDAP values.yml file with this content:

micronaut:
  security:
    # Ldap authentificaton configuration
    ldap:
      default:
        enabled: true
        context:
          server: 'ldap://ourLDAP:636'
          managerDn: "CN=name,OU=CORP,OU=USERS,DC=company,DC=CORP"
          managerPassword: 'pass'
        search:
          base: "dc=company,dc=corp"
        groups:
          enabled: true
          subtree: true
          base: "OU=ListDistribution,OU=GROUPS,DC=company,DC=corp"
          filter: 'member={0}'
...

configuration: 
  akhq:
    server:
      access-log:
        enabled: false
    security:
      default-group: reader
      groups:
        reader:
          roles:
            - topic/read
            - node/read
            - topic/data/read
            - group/read
            - registry/read
            - connect/read
      ldap:
        group:
          IT Integrations:
            groups: # Akhq groups list
              - reader

But the login page doesn't do anything. Here the network logs when we try to log:

akhq_1             | 2020-09-11 11:56:36,962 INFO  pGroup-1-5 org.akhq.log.access        [Date: 2020-09-11T11:56:36.951759Z] [Duration: 10 ms] [Url: POST /login 303] [Status: 0] [Ip: 172.18.0.1] [Length: 8080] [Port: {}]
akhq_1             | 2020-09-11 11:56:37,020 INFO  pGroup-1-2 org.akhq.log.access        [Date: 2020-09-11T11:56:37.008205Z] [Duration: 12 ms] [Url: GET /api/me 200] [Status: 574] [Ip: 172.18.0.1] [Length: 8080] [Port: {}]

How we can debuug the connection? Is correct all our configuration?

Thanks and regards,

skaravad commented 3 years ago

@gerardgorrion , did you try quotes around your AD group ? "IT Integrations" , also your LDAP server config is somewhat confusing to me, unless it was just an example, server: 'ldap://ourLDAP:636' port 636 is LDAPs (TLS) , you the URL should be server: 'ldaps://ourLDAP:636' and if you are planning secure LDAP, you should import the rootCA of your LDAP into the default JAVA truststore (cacerts).

To make things little easy during your testing, you should just try creating active directory group with no spaces (akhq-admins) , and use plain socket LDAP server: 'ldap://ourLDAP:389'

I'll try to find a some Active Directory, I think one good working config will help many.

gerardgorrion commented 3 years ago

port 636 is LDAPs (TLS) , you the URL should be server: 'ldaps://ourLDAP:636' and if you are planning secure LDAP, you should import the rootCA of your LDAP into the default JAVA truststore (cacerts).

Hi again, we connect with ldap://ourLDAP:389 via docker local, here the application.yml

micronaut:
  security:
    ldap:
      default:
        enabled: true
        context:
          server: 'ldap://ourLDAP:389'
          managerDn: 'username'
          managerPassword: 'password'
        search:
          enabled: true
          base: "OU=CORP,OU=USERS,DC=company,DC=corp"
          filter: "sAMAccountName={0}"
        groups:
          enabled: true
          subtree: true
          base: 'OU=distlist,OU=GROUPS,DC=company,DC=corp'
          filter: 'member={0}'
akhq:
  connections:
    docker-kafka-server:
      properties:
        bootstrap.servers: "kafka:9092"
      schema-registry:
        url: "http://schema-registry:8085"
      connect:
        url: "http://connect:8083"
configuration:
  akhq:
    security:
      ldap:
        group:
          groupname:
            groups:
              - admin

Now we want to login via ldaps://ourLDAP:636 We have the .cer exported via apache directory studio of our ldap. How can we add this cert to docker image to test?

Thanks!

tchiotludo commented 3 years ago

here is : https://micronaut-projects.github.io/micronaut-security/latest/guide/#io.micronaut.configuration.security.ldap.configuration.LdapConfiguration$GroupConfiguration

espacially :

-Djavax.net.ssl.trustStore="<path to truststore file>"
-Djavax.net.ssl.trustStorePassword="<passphrase for truststore>"

gerardgorrion commented 3 years ago

Thanks for the ssl information. Then we try to configure in local docker adding .cer certificate of our LDAP. How we can add .cer cert to use it inside docker image without runing a new container?

Another question, when we started the localhost:8080/ui, docker log registry show:

2020-09-17 06:52:10,195 WARN 1-thread-6 o.a.k.clients.ClientUtils Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka

2020-09-17 06:52:10,219 ERROR pGroup-1-2 o.a.c.ErrorController Failed to create new KafkaAdminClient

org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

And show a popup red with the same options (we no use kafka to test akhq and akhq can't resolve kafka:9092 obviously). Then we go to localhost:8080/ui/login and test with a ldap user:

2020-09-17 06:52:34,557 DEBUG pGroup-1-4 LdapAuthenticationProvider Starting authentication with configuration [default]

2020-09-17 06:52:34,557 DEBUG pGroup-1-4 LdapAuthenticationProvider Attempting to initialize manager context

2020-09-17 06:52:34,670 DEBUG pGroup-1-4 LdapAuthenticationProvider Manager context initialized successfully

2020-09-17 06:52:34,670 DEBUG pGroup-1-4 LdapAuthenticationProvider Attempting to authenticate with user [username]

2020-09-17 06:52:34,731 DEBUG pGroup-1-4 LdapAuthenticationProvider User found in context [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]. Attempting to bind.

2020-09-17 06:52:34,781 DEBUG pGroup-1-4 LdapAuthenticationProvider Successfully bound user [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]. Attempting to retrieving groups.

2020-09-17 06:52:34,809 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DL]

2020-09-17 06:52:34,811 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DLPais, DL]

2020-09-17 06:52:34,813 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DLHomeExternos, DLPais, DL]

2020-09-17 06:52:34,816 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DLHomeExternos, DLPais, IT, DL]

2020-09-17 06:52:34,817 DEBUG pGroup-1-4 .DefaultLdapGroupProcessor Attempting to add any additional groups...

2020-09-17 06:52:34,818 DEBUG pGroup-1-4 LdapAuthenticationProvider Group search returned [[DLHomeExternos, DLPais, IT , DL]] for user [username]

2020-09-17 06:52:34,820 TRACE pGroup-1-4 LdapAuthenticationProvider Attempting to map [username] with groups [[DLHomeExternos, DLPais, IT , DL]] to an authentication response.

2020-09-17 06:52:34,824 DEBUG pGroup-1-4 LdapAuthenticationProvider Response successfully created for [username]. Response is authenticated: [true]

But the login page doesn't change to localhost:8080/ui, and we try to navigate another time to ui and nothing happens, login page no change.

Some bug? or we need kafka to work the login page? We want to check if the roles configured are correct.

Regards,

gerardgorrion commented 3 years ago

Hi again,

The endpoints micronaut config doesn't work or there're in a bad line code?:

micronaut:
  security:
    ldap:
      default:
        enabled: true
        context:
          server: 'ldap://ourldap:389'
          managerDn: 'user'
          managerPassword: 'pass'
        search:
          enabled: true
          base: "OU=CORP,OU=USERS,DC=company,DC=corp"
          filter: "sAMAccountName={0}"
        groups:
          enabled: true
          subtree: true
          base: 'OU=distributionlist,OU=GROUPS,DC=company,DC=corp'
          filter: 'member={0}'
        endpoints:
          login:
            enabled: true
          logout:
            enabled: true
          session:
            enabled: true
            login-success-target-url: /ui
            login-failure-target-url: /ui/login/authFailed

Do you know if there're option to add anothers login button options? regards

gerardgorrion commented 3 years ago

Solved SSL connection:

-Djavax.net.ssl.trustStore=<path to truststore file>
-Djavax.net.ssl.trustStorePassword=<passphrase for truststore>

we dont need to use " ".

evaisman-tc commented 3 years ago

Adding onto this in case others have a similar issue:

If you are having trouble with the group search option, try wrapping the filter with ().

Here's what my group search looks like:

        groups:
          enabled: true
          base: "ou=Group,dc=example,dc=com"
          filter: "(memberUid=*)

tchiotludo / akhq

LDAP Config and questions #162