Use case different than sidecar

[gustavomcarmo]

Hi kafka-proxy contributors,

I'd like to know if kafka-proxy can be used in such a way different than the sidecar architecture. The project README basically mentions the use of localhost for accessing kafka-proxy, and I need it much more as a reverse proxy for accessing the backend Kafka brokers.

Thanks for any help.

[everesio]

You can configure kafka-proxy to run also on remote server e.g.

 kafka-proxy server --bootstrap-server-mapping "kafka-0.broker:9092,0.0.0.0:32401,kafka-proxy.remote:32401" \
                   --bootstrap-server-mapping "kafka-1.broker:9092,0.0.0.0:32402,kafka-proxy.remote:32402" \
                   --bootstrap-server-mapping "kafka-2.broker:9092,0.0.0.0:32403,kafka-proxy.remote:32403" \
           --dynamic-listeners-disable  

kafka-proxy.remote would be the host name where proxy runs. The kafka-proxy.remote:32501, kafka-proxy.remote:32502 and kafka-proxy.remote:32503 could be used as kafka bootstrap servers for the clients.

[gustavomcarmo]

Thanks a lot, @everesio!

I'm going now into LDAP authentication configuration, trying to figure out the right Kafka client settings to use, unfortunately with no success. Could you please give me an example of jaas.conf and other eventual configuration required for the client?

I'm eager to make it work and contribute with the kafka-proxy at least improving the documentation 😄

[everesio]

kafka proxy can intercept the sasl user/password and perform authentication against plugin e.g. LDAP.

make clean build plugin.auth-ldap && build/kafka-proxy server \
                             --auth-local-enable \
                             --auth-local-command build/auth-ldap \
                             --auth-local-param "--url=ldaps://ldap.example.com:636" \
                             --auth-local-param "--user-dn=cn=users,dc=exemple,dc=com" \
                             --auth-local-param "--user-attr=uid" \
                             --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400"

LDAP plugin will perform user bind with DN. Client jaas is not different than standard jaas used for SASL

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="alice"
  password="alice-secret";
};

[gustavomcarmo]

I'm using scripts from the bin folder of the Kafka installation (kafka-topics.sh, kafka-console-producer.sh and kafka-console-consumer.sh) for testing and I'm facing some issues. I'll provide more details about that in a new comment soon.

Have you ever used this Kafka client implementation for testing the kafka-proxy LDAP authentication?

Thanks, @everesio!

[gustavomcarmo]

Hi @everesio,

I've just created the kafka-proxy-test project with all the details regarding my tests. I appreciate any help 😃

[everesio]

1. I've just added SaslAuthenticateRequestV1, so you must use latest image with the https://github.com/grepplabs/kafka-proxy/commit/21f0ff9964772af5263fb17e184ffac143406d61.
2. Set sasl.mechanism=PLAIN in you client.properties
3. Your user has a user-attr cn not uid, you must disable tls

      - server
      - --bootstrap-server-mapping=kafka-0:9092,0.0.0.0:32400,localhost:32400
      - --bootstrap-server-mapping=kafka-1:9092,0.0.0.0:32401,localhost:32401
      - --bootstrap-server-mapping=kafka-2:9092,0.0.0.0:32402,localhost:32402
      - --dynamic-listeners-disable
      - --auth-local-enable
      - --auth-local-command=/auth-ldap
      - --auth-local-param=--start-tls=false
      - --auth-local-param=--url=ldap://openldap:389
      - --auth-local-param=--user-dn=ou=people,dc=example,dc=org
      - --auth-local-param=--user-attr=cn  

[gustavomcarmo]

Hi @everesio,

After your latest instructions, I'm still getting errors. Please take a look once more at the kafka-proxy-test.

Thanks!

[everesio]

ok. thanks for reporting. this kafka api change is not implemented yet. I will notify you when it is done

[everesio]

BTW. if you use older kafka clients e.g (kafka version 2.3), it should work

[gustavomcarmo]

Yes, you're right! Thanks a lot! Check it out here.

Isn't it something to have in the README? The Kafka client versions supported. If you don't mind, I can submit a PR with this in the documentation.

[everesio]

@gustavomcarmo please check the latest image. local auth should work with kafka 2.5 clients as well.

[gustavomcarmo]

Hi @everesio,

Now it works, thanks! When are you planning to create a new release?

BTW, the kafka-proxy LDAP authentication plugin doesn't support group filtering yet, right? This would be a nice enhancement, while not all of the LDAP users should access Kafka, but only those who would be in a specific LDAP group. What are your thoughts about that?

[everesio]

release v0.2.4 is out.

LDAP group filtering (and maybe caching) would be a nice feature, but I will not have time to implement it soon. Feel free to contribute.

[gustavomcarmo]

Hi @everesio,

I'll find some time to contribute with the LDAP group filtering feature.

BTW, has kafka-proxy some limitation in terms of number of applications connected? I've tried to connect a second application to kafka-proxy, from a different machine, with no success.

Thanks!

[gustavomcarmo]

Hi @everesio,

Is there a logo for kafka-proxy? I would like to add it to a project diagram.

Thanks!

[everesio]

Hi @gustavomcarmo

1. If you want to add LDAP group filtering feature, please create a PR.
2. There are no limitations in proxy in terms of number of applications connected. If you cannot connect from a different machine, there is probably some misconfiguration. Due kafka protocol specifics, the setup can be sometimes tricky as brokers' advertised listeners, proxy settings and client config must "match".
3. There is no logo for kafka-proxy. If you want you can propose something ;-) Kind regards, Michal