AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

Rename prefix for additional properties for KafkaStreamReader #91

Open kevinwallimann opened 4 years ago

kevinwallimann commented 4 years ago

Currently, additional properties for the KafkaStreamReader have to be specified with the prefix reader.options. E.g. reader.options.kafka.security.protocol or reader.options.kafka.ssl.key.password

This prefix is inconsistent with all other properties which start with decoder.avro., writer.parquet., manager.checkpoint or transformer.columns.. The properties for the KafkaStreamReader should start with reader.kafka., i.e. reader.kafka.options.kafka.security.protocol.

For additional properties like reader.options.failOnDataLoss or reader.options.minPartitions it's hard to find out which reader implementation they belong to.

Tasks

How to migrate

kevinwallimann commented 4 years ago

Removed milestone v3.x.0 to reduce number of breaking changes for v3.0.0. This change is not urgent as long as there is no conflict with other readers.