big-andy-coates
commented
6 years ago Conclusions so far
Kafka SSL - Full support
SSL is fully supported using standard Consumer/Producer config. There is now a test proving this and documentation, with link to the main Kafka docs.
Kafka SASL - Full support
SSL is fully supported using standard Consumer/Producer config. There is now a test proving this and documentation, with link to the main Kafka docs.
Kafka ACLs - Partial support
Kafka Clusters that use ACLs can be supported, but require the KSQL user for interactive clusters to have open access to the Kafka resources. For non-interactive KSQL clusters the KSQL user can be locked down, but determining the list of ACLs that is needed is manual / painful / non-intuitive.
It should be possible to provide tooling to list out the required ACLs for a specific statement, or sql file, which would improve things dramatically. (#948 created to track).
It's also worth noting that the SimpleAclAuthorizer that comes with AK is not ideal for KSQL / Streams, as it does not support partial-wildcard or regex resource names. So you can not, for example, set ACLs in Kafka to allow KSQL rights to produce/consume from topics "ksql*", or use consumer groups "ksql*". I think if we want better integration with Kafka clusters running ACLs, then we need a better Authorizer implementation available in the brokers. We could initially ship this with KSQL/CP, until we can get it into the AK release. (#949 created to track).
Confluent Cloud - Supported
Docs added to give users access to the settings @apurvam documented.
Schema Repo SSL - Supported (PITA)
Docs added covering how KSQL can be configured to talk to the Schema Registry over HTTPS.
However, this currently requires setting environment variables, as the SchemaRegistryClient we are using does not support receiving key store details via properties, as other Kafka components can. (#950 created to track)
The docs do not currently cover how to configure KSQL to talk to a secure Kafka cluster, (or ZK / Schema repo for that matter), nor do they cover what Kafka ACLs are needed for KSQL to 'just work', or alternative approaches.
This is causing a lot of people to run into issues when trying out KSQL against their own clusters. (There is a lot of chatter on the community slack channel on this).