Use authenticated user when sending _INFO to Kafka

AbsaOSS / enceladus

Dynamic Conformance Engine

Apache License 2.0

31 stars 14 forks source link

Use authenticated user when sending _INFO to Kafka #1807

Closed benedeki closed 3 years ago

benedeki commented 3 years ago

Background

In case Kafka requires user authentication to write into a topic, this needs to be supported by the Kafka plugin

Feature

It's to be discussed and decided if the user executing the SparkJobs would be the one authenticating against Kafka, or there would be one user set-up to do the writing. Or the former being a fallback of the later.

Zejnilovic commented 3 years ago

Release Notes: Added new properties for KafkaPlugins to be able to connect to Secured SchemaRegistry

New configs for SparkJobs using kafak plugins

kafka.schema.registry.basic.auth.credentials.source
kafka.schema.registry.basic.auth.user.info