search

ultratendency / kafka-ldap-integration

Enhancing Kafka security, PlainSaslServer with LDAP binding and SimpleAclAuthorizer with LDAP group membership
MIT License
3 stars 0 forks source link

Unable able to connect to OpenLdap server #483

Open chaslingl opened 5 months ago

chaslingl commented 5 months ago

Hi,

My environment is running docker container for OpenLDAP (osixia/openldap:1.3.0), Zookeeper, Kafka (3.6.1).

Here is error from broker server.log [2024-02-27 21:42:27,027] ERROR URI is not absolute - authentication and authorization will fail! (com.ultratendency.kafka.ldap.LDAPConfig) [2024-02-27 21:42:27,052] INFO Bind and group caches are initialized (com.ultratendency.kafka.ldap.common.LDAPCache) [2024-02-27 21:42:27,173] ERROR Authentication and authorization will fail! Exception when connecting to (,0) - null (com.ultratendency.kafka.ldap.common.LDAPBase) [2024-02-27 21:42:27,174] ERROR No LDAP connection, cannot authenticate [=uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com,, =uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com,] and related password! (com.ultratendency.kafka.ldap.authentication.LDAPAuthentication) [2024-02-27 21:42:27,176] ERROR Authentication End - authentication failed - user=uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com, status=denied (com.ultratendency.kafka.ldap.authentication.SimpleLDAPAuthentication) [2024-02-27 21:42:27,177] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /xxx.xxx.xxx.xxx (channelId=xxx.xxx.xxx.xxx:9092-xxx.xxx.xxx.xxx:53812-0) (Authentication failed: Invalid username or password) (org.apache.kafka.common.network.Selector)

Thanks.

julianbergner commented 5 months ago

Hi,

can you share your ldapconfig.yaml, please? Is the host reachable from your broker container?

chaslingl commented 5 months ago

I only change the from 636 to 389. I want to test non-secure port first.

$ cat ldapconfig.yml

host of the LDAP server

host: ldap

port of the LDAP server

port: 389

connectionTimout in milliseconds for LDAP

connTimeout: 60000

Placement of admin user in LDAP tree

adminBaseDN: ou=users,dc=security,dc=example,dc=com

Admin user attribute for DN completion

adminUid: uid

Placement of users in LDAP tree

usrBaseDN: ou=users,dc=security,dc=example,dc=com

User attribute for DN completion

usrUid: uid

Placement of groups in LDAP tree

grpBaseDN: ou=groups,dc=security,dc=example,dc=com

Group attribute for DN completion

grpUid: cn

Group membership attribute name

grpAttrName: uniqueMember

Lifetime of user entry in cache after cache-write - IN MINUTES

usrCacheExpire: 6

Lifetime of group entry in cache after cache-write - IN MINUTES

grpCacheExpire: 6

"host" doe you mean kafka broker container or my laptop? I installed pint and ldap. I can ping and query user from broker container.

Thanks.

On Thu, Feb 29, 2024 at 4:11 PM Julian Bergner @.***> wrote:

Hi,

can you share your ldapconfig.yaml, please? Is the host reachable from your broker container?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1972057017, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FOSDWEM65YGPYIRUQLYV6TRNAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSGA2TOMBRG4 . You are receiving this because you authored the thread.Message ID: @.***>

julianbergner commented 5 months ago

With host I meant the variable in the yaml with your value ldap. Can you paste your ldapsearch command from your broker container, please?

Edit: Can the user which runs the broker access ldapconfig.yaml? What are the file permissions?

chaslingl commented 5 months ago

'ldap' is the hostname of ldap container.

I ran the following ldapsearch command in 'broker' container @.***:/usr/local/kafka# ldapsearch -x -H ldap://ldap:389 -b dc=example,dc=com -D "uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com" -w broker

extended LDIF

#

LDAPv3

base with scope subtree

filter: (objectclass=*)

requesting: ALL

#

search result

search: 2 result: 32 No such object

numResponses: 1

Thanks.

On Thu, Feb 29, 2024 at 5:32 PM Julian Bergner @.***> wrote:

With host I meant the variable in the yaml with your value ldap. Can you paste your ldapsearch command from your broker container, please?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1972147982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FKTSZ4UYCDMFYVUIJTYV65BBAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSGE2DOOJYGI . You are receiving this because you authored the thread.Message ID: @.***>

julianbergner commented 5 months ago

Can the user which runs the broker access ldapconfig.yaml? What are the file permissions?

chaslingl commented 5 months ago

Yes, container is running by root. root@broker:/usr/local/kafka# ls -la config/ldapconfig.yml -rwxr-xr-x 1 root root 773 Feb 27 21:30 config/ldapconfig.yml

chaslingl commented 5 months ago

Yes, the container is running by root. @.***:/usr/local/kafka# ls -la config/ldapconfig.yml -rwxr-xr-x 1 root root 773 Feb 27 21:30 config/ldapconfig.yml

On Fri, Mar 1, 2024 at 4:19 AM Julian Bergner @.***> wrote:

Can the user which runs the broker access ldapconfig.yaml ? What are the file permissions?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1972913352, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FN5VF3OBDCMQCVMHFTYWBI3HAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSHEYTGMZVGI . You are receiving this because you authored the thread.Message ID: @.***>

chaslingl commented 5 months ago

Hi,

Any suggestions that I can try?

Thanks

On Fri, Mar 1, 2024 at 9:19 AM Ccc L @.***> wrote:

Yes, the container is running by root. @.***:/usr/local/kafka# ls -la config/ldapconfig.yml -rwxr-xr-x 1 root root 773 Feb 27 21:30 config/ldapconfig.yml

On Fri, Mar 1, 2024 at 4:19 AM Julian Bergner @.***> wrote:

Can the user which runs the broker access ldapconfig.yaml ? What are the file permissions?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1972913352, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FN5VF3OBDCMQCVMHFTYWBI3HAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSHEYTGMZVGI . You are receiving this because you authored the thread.Message ID: @.***>

julianbergner commented 5 months ago

Hi,

yes I assume that the ldapconfig.yml is not read correctly. How do you start your broker service and how do you set the Classpath?

chaslingl commented 5 months ago

Hi,

I placed the ldapconfig.yml under /usr/local/kafak/config and set the path in the CLASSPATH when I builded docker image.

I start the broker with ==> bin/kafka-server-start.sh config/server.properties

Thanks,

On Mon, Mar 4, 2024 at 4:57 PM Julian Bergner @.***> wrote:

Hi,

yes I assume that the ldapconfig.yml is not read correctly. How do you start your broker service and how do you set the Classpath?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1977613606, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FKESLCVROHS5XQV3B3YWT34HAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZXGYYTGNRQGY . You are receiving this because you authored the thread.Message ID: @.***>

julianbergner commented 5 months ago

can you share your docker file please?

chaslingl commented 5 months ago

FROM ubuntu:22.04

RUN apt-get update

Kafka servers are not handling http protocol natively. To properly set a

healthcheck up, netcat is required RUN apt-get install -y \ openjdk-17-jdk \ wget curl && \ rm -rf /var/lib/apt/lists/*

ENV KAFKA_PATH=/usr/local/kafka

RUN wget https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz && \ tar xzf kafka_2.13-3.6.1.tgz && \ mv kafka_2.13-3.6.1 $KAFKA_PATH && \

Remove the compressed file and save around 90 Mb

rm -rf kafka_2.13-3.6.1.tgz

WORKDIR $KAFKA_PATH USER root

...

Set plugin in CLASSPATH

ENV CLASSPATH="$KAFKA_PATH/plugins/*:$KAFKA_PATH/config:$CLASSPATH" RUN echo $CLASSPATH

Here is the basic stuff, and I removed any reference add files.

Thanks.

On Wed, Mar 6, 2024 at 6:54 AM Julian Bergner @.***> wrote:

can you share your docker file please?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1980807830, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FNQS5EZUVA7NUERTRTYW4G2FAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQHAYDOOBTGA . You are receiving this because you authored the thread.Message ID: @.***>

chaslingl commented 5 months ago

Hi,

Here are these errors [2024-03-06 18:36:48,303] ERROR URI is not absolute - authentication and authorization will fail! (com.ultratendency.kafka.ldap.LDAPConfig) [2024-03-06 18:36:48,336] INFO Bind and group caches are initialized (com.ultratendency.kafka.ldap.common.LDAPCache) [2024-03-06 18:36:48,450] ERROR Authentication and authorization will fail! Exception when connecting to (,0) - null (com.ultratendency.kafka.ldap.common.LDAPBase) [2024-03-06 18:36:48,452] ERROR No LDAP connection, cannot authenticate [=srvkafkabroker,, =srvkafkabroker,] and related password! (com.ultratendency.kafka.ldap.authentication.LDAPAuthentication) [2024-03-06 18:36:48,453] ERROR Authentication End - authentication failed

I don't know if the code cannot find ldapconfig.yml file, or the ldap connection can't establish, or this is an invalid user or password. The ldapconfig.yml contents are from ultratendency/kafka-ldap-integration.

@.:/usr/local/kafka# echo $CLASSPATH /usr/local/kafka/plugins/::/usr/local/kafka/config **@.***:/usr/local/kafka# ls /usr/local/kafka/config/ldapconfig.yml /usr/local/kafka/config/ldapconfig.yml

Thanks

On Wed, Mar 6, 2024 at 11:02 AM Ccc L @.***> wrote:

FROM ubuntu:22.04

RUN apt-get update

Kafka servers are not handling http protocol natively. To properly set a

healthcheck up, netcat is required RUN apt-get install -y \ openjdk-17-jdk \ wget curl && \ rm -rf /var/lib/apt/lists/*

ENV KAFKA_PATH=/usr/local/kafka

RUN wget https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz && \ tar xzf kafka_2.13-3.6.1.tgz && \ mv kafka_2.13-3.6.1 $KAFKA_PATH && \

Remove the compressed file and save around 90 Mb

rm -rf kafka_2.13-3.6.1.tgz

WORKDIR $KAFKA_PATH USER root

...

Set plugin in CLASSPATH

ENV CLASSPATH="$KAFKA_PATH/plugins/*:$KAFKA_PATH/config:$CLASSPATH" RUN echo $CLASSPATH

Here is the basic stuff, and I removed any reference add files.

Thanks.

On Wed, Mar 6, 2024 at 6:54 AM Julian Bergner @.***> wrote:

can you share your docker file please?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1980807830, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FNQS5EZUVA7NUERTRTYW4G2FAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQHAYDOOBTGA . You are receiving this because you authored the thread.Message ID: @.***>

chaslingl commented 4 months ago

Any suggestions about my issue.

Thanks.

On Wed, Mar 6, 2024 at 12:54 PM Ccc L @.***> wrote:

Hi,

Here are these errors [2024-03-06 18:36:48,303] ERROR URI is not absolute - authentication and authorization will fail! (com.ultratendency.kafka.ldap.LDAPConfig) [2024-03-06 18:36:48,336] INFO Bind and group caches are initialized (com.ultratendency.kafka.ldap.common.LDAPCache) [2024-03-06 18:36:48,450] ERROR Authentication and authorization will fail! Exception when connecting to (,0) - null (com.ultratendency.kafka.ldap.common.LDAPBase) [2024-03-06 18:36:48,452] ERROR No LDAP connection, cannot authenticate [=srvkafkabroker,, =srvkafkabroker,] and related password! (com.ultratendency.kafka.ldap.authentication.LDAPAuthentication) [2024-03-06 18:36:48,453] ERROR Authentication End - authentication failed

  • user=srvkafkabroker, status=denied (com.ultratendency.kafka.ldap.authentication.SimpleLDAPAuthentication)

I don't know if the code cannot find ldapconfig.yml file, or the ldap connection can't establish, or this is an invalid user or password. The ldapconfig.yml contents are from ultratendency/kafka-ldap-integration.

@.:/usr/local/kafka# echo $CLASSPATH /usr/local/kafka/plugins/::/usr/local/kafka/config **@.***:/usr/local/kafka# ls /usr/local/kafka/config/ldapconfig.yml /usr/local/kafka/config/ldapconfig.yml

Thanks

On Wed, Mar 6, 2024 at 11:02 AM Ccc L @.***> wrote:

FROM ubuntu:22.04

RUN apt-get update

Kafka servers are not handling http protocol natively. To properly set

a healthcheck up, netcat is required RUN apt-get install -y \ openjdk-17-jdk \ wget curl && \ rm -rf /var/lib/apt/lists/*

ENV KAFKA_PATH=/usr/local/kafka

RUN wget https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz && \ tar xzf kafka_2.13-3.6.1.tgz && \ mv kafka_2.13-3.6.1 $KAFKA_PATH && \

Remove the compressed file and save around 90 Mb

rm -rf kafka_2.13-3.6.1.tgz

WORKDIR $KAFKA_PATH USER root

...

Set plugin in CLASSPATH

ENV CLASSPATH="$KAFKA_PATH/plugins/*:$KAFKA_PATH/config:$CLASSPATH" RUN echo $CLASSPATH

Here is the basic stuff, and I removed any reference add files.

Thanks.

On Wed, Mar 6, 2024 at 6:54 AM Julian Bergner @.***> wrote:

can you share your docker file please?

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1980807830, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FNQS5EZUVA7NUERTRTYW4G2FAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQHAYDOOBTGA . You are receiving this because you authored the thread.Message ID: @.***>

julianbergner commented 4 months ago

In fact, no. I cannot reproduce your behaviour. When I am using your ldapconfig.yml and host an openldap container I am getting this log:

[2024-03-08 15:48:08,742] INFO file:/etc/kafka/ldapconfig.yaml read (com.ultratendency.kafka.ldap.LDAPConfig) [2024-03-08 15:48:08,742] INFO LDAPConfig for classpath is cached (com.ultratendency.kafka.ldap.LDAPConfig) [2024-03-08 15:48:08,751] INFO ldap configuration values: Config(host=ldap, port=389, connTimeout=60000, adminBaseDN=ou=users,dc=security,dc=example,dc=com, adminUid=uid, usrBaseDN=ou=users,dc=security,dc=example,dc=com, usrUid=uid, grpBaseDN=ou=groups,dc=security,dc=example,dc=com, grpUid=cn, grpAttrName=uniqueMember, usrCacheExpire=6, grpCacheExpire=6) (com.ultratendency.kafka.ldap.LDAPConfig) [2024-03-08 15:48:08,765] INFO Bind and group caches are initialized (com.ultratendency.kafka.ldap.common.LDAPCache)

julianbergner commented 4 months ago

'ldap' is the hostname of ldap container. I ran the following ldapsearch command in 'broker' container @.:/usr/local/kafka# ldapsearch -x -H ldap://ldap:389 -b dc=example,dc=com -D "uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com" -w broker # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=) # requesting: ALL # # search result search: 2 result: 32 No such object # numResponses: 1 Thanks. … On Thu, Feb 29, 2024 at 5:32 PM Julian Bergner **@.> wrote: With host I meant the variable in the yaml with your value ldap. Can you paste your ldapsearch command from your broker container, please? — Reply to this email directly, view it on GitHub <#483 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FKTSZ4UYCDMFYVUIJTYV65BBAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSGE2DOOJYGI . You are receiving this because you authored the thread.Message ID: @.>

Btw: This ldapsearch throws an error. Shouldn't be related to the wrong ldap host, but still you have an ldap issues even without the library involved.

chaslingl commented 4 months ago

I fixed the ldapsearch search baseDN. I can get the user on broker container

$ docker exec -it broker /bin/bash @.***:/usr/local/kafka# ldapsearch -x -H ldap://ldap:389 -b uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com -D "uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com" -w broker

extended LDIF

#

LDAPv3

base with

scope subtree

filter: (objectclass=*)

requesting: ALL

#

srvkafkabroker, users, security.example.com

dn: uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com objectClass: top objectClass: inetOrgPerson objectClass: person objectClass: organizationalPerson cn: Kafka Broker sn: Broker uid: srvkafkabroker userPassword:: YnJva2Vy

search result

search: 2 result: 0 Success

numResponses: 2

numEntries: 1

Thanks.

On Fri, Mar 8, 2024 at 10:18 AM Julian Bergner @.***> wrote:

'ldap' is the hostname of ldap container. I ran the following ldapsearch command in 'broker' container @.

*:/usr/local/kafka# ldapsearch -x -H ldap://ldap:389 -b dc=example,dc=com -D "uid=srvkafkabroker,ou=users,dc=security,dc=example,dc=com" -w broker # extended LDIF # # LDAPv3 # base with scope subtree # filter: (objectclass=) # requesting: ALL # # search result search: 2 result: 32 No such object # numResponses: 1 Thanks. … <#m-8949780311591335652> On Thu, Feb 29, 2024 at 5:32 PM Julian Bergner @.*> wrote: With host I meant the variable in the yaml with your value ldap. Can you paste your ldapsearch command from your broker container, please? — Reply to this email directly, view it on GitHub <#483 (comment) https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1972147982>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FKTSZ4UYCDMFYVUIJTYV65BBAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSGE2DOOJYGI https://github.com/notifications/unsubscribe-auth/AQVN3FKTSZ4UYCDMFYVUIJTYV65BBAVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSGE2DOOJYGI . You are receiving this because you authored the thread.Message ID: @. **>

Btw: This ldapsearch throws an error. Shouldn't be related to the wrong ldap host, but still you have an ldap issues even without the library involved.

— Reply to this email directly, view it on GitHub https://github.com/ultratendency/kafka-ldap-integration/issues/483#issuecomment-1985985154, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQVN3FK22VIFL7XCVNWP4GDYXHQG3AVCNFSM6AAAAABD6FFPE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBVHE4DKMJVGQ . You are receiving this because you authored the thread.Message ID: @.***>