Closed jessetandor closed 4 years ago
Hi,
First of all, do you use the dev version ? the release is not done yet for now (coming soon).
For the question :
io.micronaut.configuration.security.ldap
to have some insight. Here is a minimal configuration with ldap & basic auth :
micronaut:
security:
enabled: true
ldap:
default:
enabled: true
context:
server: 'ldap://ldap.forumsys.com:389'
managerDn: 'cn=read-only-admin,dc=example,dc=com'
managerPassword: 'password'
search:
base: "dc=example,dc=com"
groups:
enabled: true
base: "dc=example,dc=com"
kafkahq:
server:
connections:
plaintext:
properties:
bootstrap.servers: "kafka:9092"
schema-registry:
url: "http://schema-registry:8085"
basic-auth-username: test_user
basic-auth-password: test_pass
connect:
url: "http://connect:8083"
basic-auth-username: test_user
basic-auth-password: test_pass
ksql:
url: "http://ksql:8088"
security:
basic-auth:
user:
password: d74ff0ee8da3b9806b18c877dbf29bbde50b5bd8e4dad7a3a725000feb82e8f1
groups:
- admin
ldap:
group:
mathematicians:
groups:
- reader
scientists:
groups:
- admin
user / pass for basicauth user & pass from here for ldap
Yap - running Dev version from Docker. Making progress but run into a small issue with AD Cert
LdapAuthenticationProvider Failed to create manager context. Returning unknown authentication failure. Encountered {}
`javax.naming.CommunicationException: simple bind failed: local.xx.xx.xx:xx`
I tried :
$ curl -i -X POST -H "Content-Type: application/json" \
> -d '{ "configuredLevel": βFALSEβ }' \
> http://localhost:5000/Dcom.sun.net.ssl.checkRevocation
HTTP/1.1 401 Unauthorized
Date: Mon, 9 Dec 2019 13:49:08 GMT
transfer-encoding: chunked
connection: close
But the result is 401. Most likely this method is not supported anymore as it's very old (https://tinyurl.com/y4hvdgfp)
Adding relevant sections below:
micronaut:
security:
# Ldap authentificaton configuration
ldap:
default:
enabled: true
context:
server: 'ldaps://xxx.xx.xxx.xxx:xxx'
managerDn: 'cn=xxx,dc=xx,dc=xx,dc=xx'
managerPassword: '123'
search:
base: "dc=company,dc=net"
groups:
enabled: true
base: "dc=company,dc=net"
# Ldap Groups configuration (when using ldap)
ldap:
group:
kafkahq-admin: # ldap group name
groups: # KafkaHQ groups list
- admin
kafkahq-readonly:
groups:
- readonly
To be honest, I don't know anything about ldaps, never used myself.
In general, in java, the certificate is mandatory in java, and you will need a custom keystore with your certificate in it : http://roufid.com/java-ldap-ssl-authentication/#1-_register_your_ssl_certificate_into_java_keystore.
I don't know if the -Dcom.sun.net.ssl.checkRevocation=false
is still valid but it's a java opts.
You need to pass it to docker image with env vars, example here for docker-compose
environment:
JAVA_OPTS: "-Dcom.sun.net.ssl.checkRevocation=false"
Yap, that helped tchiotludo. I was able to catch logs and eventually no Cert needed to be set for the managerDN login. Your DN has to be 100% correct ofc, mine was slightly off.
I have run into another issue though, maybe you know but AD power users might know more.
Ive created two Groups
distinguishedName: CN=kafkahq-readonly,OU=Kafka,OU=company Groups,DC=ad,DC=company,DC=net
distinguishedName: CN=kafkahq-admin,OU=Kafka,OU=company Groups,DC=ad,DC=company,DC=net
And called them in my LDAP section
# Ldap Groups configuration (when using ldap)
ldap:
group:
kafkahq-admin: # ldap group name
groups: # KafkaHQ groups list
- admin
kafkahq-readonly:
groups:
- readonly
However when trying to login ( a login screen is presented so the connection is there ) - i get
2019-12-11 11:47:19,273 DEBUG pGroup-1-2 LdapAuthenticationProvider Starting authentication with configuration [default]
2019-12-11 11:47:19,273 DEBUG pGroup-1-2 LdapAuthenticationProvider Attempting to initialize manager context
2019-12-11 11:47:20,372 DEBUG pGroup-1-2 LdapAuthenticationProvider Manager context initialized successfully
2019-12-11 11:47:20,373 DEBUG pGroup-1-2 LdapAuthenticationProvider Attempting to authenticate with user [user]
2019-12-11 11:47:20,537 DEBUG pGroup-1-2 LdapAuthenticationProvider User found in context [CN=user,OU=company Users,DC=ad,DC=company,DC=net]. Attempting to bind.
2019-12-11 11:47:21,041 DEBUG pGroup-1-2 LdapAuthenticationProvider Successfully bound user [CN=user,OU=company Users,DC=ad,DC=company,DC=net]. Attempting to retrieving groups.
2019-12-11 11:47:21,183 DEBUG pGroup-1-2 .DefaultLdapGroupProcessor No groups found!
2019-12-11 11:47:21,184 DEBUG pGroup-1-2 .DefaultLdapGroupProcessor Attempting to add any additional groups...
2019-12-11 11:47:21,184 DEBUG pGroup-1-2 LdapAuthenticationProvider Group search returned [[]] for user [user]
2019-12-11 11:47:21,185 TRACE pGroup-1-2 LdapAuthenticationProvider Attempting to map [user] with groups [[]] to an authentication response.
2019-12-11 11:47:21,188 DEBUG pGroup-1-2 LdapAuthenticationProvider Response successfully created for [user]. Response is authenticated: [true]
So the manager login is succesful, the user credentials are also correct as the user is bound, but no groups are found for the user thus i get Forbidden on the web side.
I cannot figure our why the groups aren't detected : i cant see wrong with my settings either:
micronaut:
security:
# Ldap authentificaton configuration
ldap:
default:
enabled: true
context:
server: 'ldaps://local.ad.company.net:636'
managerDn: 'CN=users,OU=xxx,OU=xxxx,DC=xx,DC=xxx,DC=xxx'
managerPassword: 'password_1234'
search:
base: "OU=xxx,DC=xx,DC=xx,DC=xx"
groups:
enabled: true
base: "OU=xxx,DC=xx,DC=xx,DC=xx"
As I see on code, the group base configuration don't return any things. So it's not you kafkahq group that failed, but it was the ldap group that is not found if it can help .
Do you have try with test ldap server first ?
micronaut:
security:
enabled: true
ldap:
default:
enabled: true
context:
server: 'ldap://ldap.forumsys.com:389'
managerDn: 'cn=read-only-admin,dc=example,dc=com'
managerPassword: 'password'
search:
base: "dc=example,dc=com"
groups:
enabled: true
base: "dc=example,dc=com"
to make it work and adapt it for your local company ad ?
Yes, the two LDAP groups arent' found, even though my user is a member of the admin group.
Based on your example if i change the search base to just : "DC=ad,DC=company,DC=net", it fails to authenticate the user logging in.
'2019-12-11 13:46:04,553 DEBUG pGroup-1-2 LdapAuthenticationProvider Failed to authenticate with user {}
javax.naming.PartialResultException: Unprocessed Continuation Reference(s)
And yes, i've been testing with multiple AD setup's π
The configuration seems to be good.
Seems that your user is not in any groups on AD.
Are you sure that you have any group with user tandor
?
The log with no groups is coming from micronaut before mapping from KafkaHQ, so must be the group lookup that failed with groups.base
configuration.
The only thing I see is that user and group is not on the same hierarchy but unfortunately, I'm not a boss in AD at all so hard to help. Sorry ...
Yes, my user is definitely part of the kafka group in AD
Are there any extra arguments we can pass in the group's LDAP section? Like filter or matchGroup ? We've had to do this before with Cerebro and that seemed to work.
all the option are here : https://micronaut-projects.github.io/micronaut-security/latest/guide/#ldap
There is option like filter for your use case, maybe on this will be working ?
Yep, so we gave up on this - it doesnt seem to search nested values, thus no groups are found. We also tried to hack it alil from micronaut's side : 1) Using Member/Memberof attributes 2) Direct Filters to narrow down the group detection (directly calling CN values)
Sadly these still detected no groups from Samba AD server.
Going back to Read Only mode: @tchiotludo Did you remove it from version 0.11.0 and up?
My config file works in read-only mode for 0.10.0 but in 0.11.0, i can see the button for topic creation...again it's the same file and syntax is correct.
kafkahq:
connections:
test:
properties:
bootstrap.servers: "x.x.x.x:9092" # IP and Port of one machine in the cluster for that region
security.protocol: SASL_PLAINTEXT # This is our setup in Kafka
sasl.mechanism: SCRAM-SHA-256 # This is our setup in Kafka
sasl.jaas.config: "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"xxxx\";" # This has to match the user creaked in Kafka
pagination:
page-size: 10 # number of elements per page
threads: 64 # Number of parallel threads to resolve page - can be performance heavy
security:
default-roles:
- topic/read
- node/read
- topic/data/read
- group/read
- registry/read
- connect/read
The configuration have change : https://github.com/tchiotludo/kafkahq/blob/a6ebf24f8c0ac2936397361aaf4ff4d3d4f706c4/application.example.yml#L105
Here is an example to have a reader by default and 1 user with all acls :
security:
default-groups: # Default groups for all the user even unlogged user
- reader
basic-auth:
user:
password: d74ff0ee8da3b9806b18c877dbf29bbde50b5bd8e4dad7a3a725000feb82e8f1
groups:
- admin
We are up and running with 0.11.0. Thanks alot..
You mentioned Multi cluster
in the features - what do you mean by this? As far as I've read, one instance of kafkaHQ can only connect to one or many machine's in the same kafka cluster. Or is that wrong?
About the ko - mo - go values for weights, what do they mean - log dir size?
Glad to know it's working, but some kind of frustrated that you can't have the ldap running.
Multi cluster mean that you can connect to multiple Kafka cluster on the same KafkaHQ instance (like dev, prod, ...).
Just duplicate the connections
on the configuration files to have multiple cluster.
Weight is log dir size, maybe should rename the cold π
Closing this one, since it's more discussion !
Feel free to add a PR on #161 π
@jessetandor @tchiotludo may be you have already got this working, I ran into similar issue and I was able to fix it. First of all, it all depends on the type of LDAP you use, the default micronaut config for group search filter is set to "uniquemember={0}" which might not hold true for Active Directory , from my vague memory it could be member (so the setting will be "member ={0}"
Here is the config section under micronaut
micronaut:
security:
ldap:
default:
enabled: true
context:
server: 'ldaps://ldapserver:636'
managerDn: 'CN=xxx,CN=Users,DC=MyDomain,DC=com"
managerPassword: 'xxxxxxxx'
search:
base: "CN=Users,DC=MyDomain,DC=com"
groups:
enabled: true
subtree: true # Pay attention here, this is needed.
base: "CN=Users,DC=MyDomain,DC=com"
filter: 'member={0}'`
Under the akhq section,
security:
default-group: reader # Default groups for all the user even unlogged user
# Groups definition
groups:
reader:
roles: ""
admin: # Group name
roles: # roles for the group
- topic/read
- topic/insert
- topic/delete
- topic/config/update
- node/read
- node/config/update
- topic/data/read
- topic/data/insert
- topic/data/delete
- group/read
- group/delete
- group/offsets/update
- registry/read
- registry/insert
- registry/update
- registry/delete
- registry/version/delete
- acls/read
- connect/read
- connect/insert
- connect/update
- connect/delete
- connect/state/update
ldap:
group:
ldapAkHq-admins: # This is your active directory group name just the part of the CN
groups:
- admin
basic-auth:
thanks @skaravad for help :)
Hi skaravad,
I was tryed to configurate my LDAP values.yml file with this content:
micronaut:
security:
# Ldap authentificaton configuration
ldap:
default:
enabled: true
context:
server: 'ldap://ourLDAP:636'
managerDn: "CN=name,OU=CORP,OU=USERS,DC=company,DC=CORP"
managerPassword: 'pass'
search:
base: "dc=company,dc=corp"
groups:
enabled: true
subtree: true
base: "OU=ListDistribution,OU=GROUPS,DC=company,DC=corp"
filter: 'member={0}'
...
configuration:
akhq:
server:
access-log:
enabled: false
security:
default-group: reader
groups:
reader:
roles:
- topic/read
- node/read
- topic/data/read
- group/read
- registry/read
- connect/read
ldap:
group:
IT Integrations:
groups: # Akhq groups list
- reader
But the login page doesn't do anything. Here the network logs when we try to log:
akhq_1 | 2020-09-11 11:56:36,962 INFO pGroup-1-5 org.akhq.log.access [Date: 2020-09-11T11:56:36.951759Z] [Duration: 10 ms] [Url: POST /login 303] [Status: 0] [Ip: 172.18.0.1] [Length: 8080] [Port: {}]
akhq_1 | 2020-09-11 11:56:37,020 INFO pGroup-1-2 org.akhq.log.access [Date: 2020-09-11T11:56:37.008205Z] [Duration: 12 ms] [Url: GET /api/me 200] [Status: 574] [Ip: 172.18.0.1] [Length: 8080] [Port: {}]
How we can debuug the connection? Is correct all our configuration?
Thanks and regards,
@gerardgorrion , did you try quotes around your AD group ? "IT Integrations"
, also your LDAP server config is somewhat confusing to me, unless it was just an example, server: 'ldap://ourLDAP:636'
port 636 is LDAPs (TLS) , you the URL should be server: 'ldaps://ourLDAP:636'
and if you are planning secure LDAP, you should import the rootCA of your LDAP into the default JAVA truststore (cacerts).
To make things little easy during your testing, you should just try creating active directory group with no spaces (akhq-admins) , and use plain socket LDAP server: 'ldap://ourLDAP:389'
I'll try to find a some Active Directory, I think one good working config will help many.
port 636 is LDAPs (TLS) , you the URL should be
server: 'ldaps://ourLDAP:636'
and if you are planning secure LDAP, you should import the rootCA of your LDAP into the default JAVA truststore (cacerts).
Hi again, we connect with ldap://ourLDAP:389 via docker local, here the application.yml
micronaut:
security:
ldap:
default:
enabled: true
context:
server: 'ldap://ourLDAP:389'
managerDn: 'username'
managerPassword: 'password'
search:
enabled: true
base: "OU=CORP,OU=USERS,DC=company,DC=corp"
filter: "sAMAccountName={0}"
groups:
enabled: true
subtree: true
base: 'OU=distlist,OU=GROUPS,DC=company,DC=corp'
filter: 'member={0}'
akhq:
connections:
docker-kafka-server:
properties:
bootstrap.servers: "kafka:9092"
schema-registry:
url: "http://schema-registry:8085"
connect:
url: "http://connect:8083"
configuration:
akhq:
security:
ldap:
group:
groupname:
groups:
- admin
Now we want to login via ldaps://ourLDAP:636 We have the .cer exported via apache directory studio of our ldap. How can we add this cert to docker image to test?
Thanks!
espacially :
-Djavax.net.ssl.trustStore="<path to truststore file>"
-Djavax.net.ssl.trustStorePassword="<passphrase for truststore>"
Thanks for the ssl information. Then we try to configure in local docker adding .cer certificate of our LDAP. How we can add .cer cert to use it inside docker image without runing a new container?
Another question, when we started the localhost:8080/ui, docker log registry show:
2020-09-17 06:52:10,195 WARN 1-thread-6 o.a.k.clients.ClientUtils Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
2020-09-17 06:52:10,219 ERROR pGroup-1-2 o.a.c.ErrorController Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
And show a popup red with the same options (we no use kafka to test akhq and akhq can't resolve kafka:9092 obviously). Then we go to localhost:8080/ui/login and test with a ldap user:
2020-09-17 06:52:34,557 DEBUG pGroup-1-4 LdapAuthenticationProvider Starting authentication with configuration [default]
2020-09-17 06:52:34,557 DEBUG pGroup-1-4 LdapAuthenticationProvider Attempting to initialize manager context
2020-09-17 06:52:34,670 DEBUG pGroup-1-4 LdapAuthenticationProvider Manager context initialized successfully
2020-09-17 06:52:34,670 DEBUG pGroup-1-4 LdapAuthenticationProvider Attempting to authenticate with user [username]
2020-09-17 06:52:34,731 DEBUG pGroup-1-4 LdapAuthenticationProvider User found in context [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]. Attempting to bind.
2020-09-17 06:52:34,781 DEBUG pGroup-1-4 LdapAuthenticationProvider Successfully bound user [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]. Attempting to retrieving groups.
2020-09-17 06:52:34,809 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DL]
2020-09-17 06:52:34,811 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DLPais, DL]
2020-09-17 06:52:34,813 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DLHomeExternos, DLPais, DL]
2020-09-17 06:52:34,816 TRACE pGroup-1-4 .DefaultLdapGroupProcessor The following groups were found for [CN=username,OU=CORP,OU=USERS,DC=company,DC=corp]: [DLHomeExternos, DLPais, IT, DL]
2020-09-17 06:52:34,817 DEBUG pGroup-1-4 .DefaultLdapGroupProcessor Attempting to add any additional groups...
2020-09-17 06:52:34,818 DEBUG pGroup-1-4 LdapAuthenticationProvider Group search returned [[DLHomeExternos, DLPais, IT , DL]] for user [username]
2020-09-17 06:52:34,820 TRACE pGroup-1-4 LdapAuthenticationProvider Attempting to map [username] with groups [[DLHomeExternos, DLPais, IT , DL]] to an authentication response.
2020-09-17 06:52:34,824 DEBUG pGroup-1-4 LdapAuthenticationProvider Response successfully created for [username]. Response is authenticated: [true]
But the login page doesn't change to localhost:8080/ui, and we try to navigate another time to ui and nothing happens, login page no change.
Some bug? or we need kafka to work the login page? We want to check if the roles configured are correct.
Regards,
Hi again,
The endpoints micronaut config doesn't work or there're in a bad line code?:
micronaut:
security:
ldap:
default:
enabled: true
context:
server: 'ldap://ourldap:389'
managerDn: 'user'
managerPassword: 'pass'
search:
enabled: true
base: "OU=CORP,OU=USERS,DC=company,DC=corp"
filter: "sAMAccountName={0}"
groups:
enabled: true
subtree: true
base: 'OU=distributionlist,OU=GROUPS,DC=company,DC=corp'
filter: 'member={0}'
endpoints:
login:
enabled: true
logout:
enabled: true
session:
enabled: true
login-success-target-url: /ui
login-failure-target-url: /ui/login/authFailed
Do you know if there're option to add anothers login button options? regards
Solved SSL connection:
-Djavax.net.ssl.trustStore=<path to truststore file>
-Djavax.net.ssl.trustStorePassword=<passphrase for truststore>
we dont need to use " ".
Adding onto this in case others have a similar issue:
If you are having trouble with the group search option, try wrapping the filter with ()
.
Here's what my group search looks like:
groups:
enabled: true
base: "ou=Group,dc=example,dc=com"
filter: "(memberUid=*)
Alil difficulty with LDAP on our end - here is our config :
Trace has been enabled in the output and we noticed the LDAP server was being contacted but on port 9092..
1) Is it expected that the kafka cluster is also setup using LDAP? 2) If not - how does KafkaHQ request a login screen? 3) Are there any other way's to debug LDAP side? 4) Any plans to support LDAPS?