search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
122 stars 1.04k forks source link

how can i use ksqldb connect to a kafka cluster which auth with sasl(kerberos) #6306

Open chanRoot opened 4 years ago

chanRoot commented 4 years ago

when i use a kafka with kerberos auth, it offers two ways for me. keytab files or plaintext both associated to a principal created by kerberos. now i want use ksqldb with it,i find docs said that i can override LoginModule whith others, does it means i can replace the value of sasl.jaas.config with Krb5LoginModule and use keytab=.. to point a keytab file? and i find the project update a API ksqldb-api-client:Client for users to set ksqldb_host and ksql_port to connect a ksqldb-server ,is there some way i can bring kerberos auth params?

chanRoot commented 4 years ago

this is my docker run-scripts, has mistakes

docker run -d \
  -p 127.0.0.1:8088:8088 \
  --add-host kafka.example.org:192.168.6.150 \
  --add-host stone.example.org:192.168.3.115 \
  -v /etc/krb5.conf:/etc/krb5.conf \
  -e KSQL_BOOTSTRAP_SERVERS=kafka.example.org:9092 \
  -e KSQL_LISTENERS=http://0.0.0.0:8088/ \
  -e KSQL_KSQL_SERVICE_ID=default_ \
  -e KSQL_KSQL_SINK_REPLICAS=3 \
  -e KSQL_KSQL_STREAMS_REPLICATION_FACTOR=3 \
  -e KSQL_KSQL_INTERNAL_TOPIC_REPLICAS=3 \
  -e KSQL_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e KSQL_SASL_MECHANISM=GSSAPI \
  -e KSQL_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=stone/stone.example.org@EXAMPLE.ORG password=stone;" \
  confluentinc/ksqldb-server:0.11.0
chanRoot commented 4 years ago

it's error log, i know KafkaAdminClient use a Properties or Map to collect some auth params and others, eg. bootstrap.server sasl.mechanism ..

[2020-09-28 05:47:27,261] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:66)
java.lang.RuntimeException: Failed to get Kafka cluster information
    at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:104)
    at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:571)
    at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:118)
    at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:73)
    at org.apache.kafka.streams.processor.internals.DefaultKafkaClientSupplier.getAdmin(DefaultKafkaClientSupplier.java:41)
    at io.confluent.ksql.services.ServiceContextFactory.lambda$create$1(ServiceContextFactory.java:58)
    at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:164)
    at io.confluent.ksql.services.DefaultServiceContext$MemoizedSupplier.get(DefaultServiceContext.java:163)
    at io.confluent.ksql.services.DefaultServiceContext.getAdminClient(DefaultServiceContext.java:108)
    at io.confluent.ksql.services.LazyServiceContext.getAdminClient(LazyServiceContext.java:33)
    at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:99)
    ... 3 more
Caused by: java.lang.IllegalArgumentException: JAAS config entry not terminated by semi-colon
    at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:121)
    at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)
    at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
    at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:134)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:481)
chanRoot commented 4 years ago

or i need to set jvm params to point files such as jaas.conf and krb5.conf?

AlanConfluent commented 4 years ago

I think you generally want something like:

-e  KSQL_SASL_JAAS_CONFIG="com.sun.security.auth.module.Krb5LoginModule required useKeyTab=... storeKey=... principal="blah";" \

to use Krb5LoginModule. More details on this module are here: https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html