AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

Use prefix writer.parquet.option for extra configuration to the parquet writers #67

Closed kevinwallimann closed 4 years ago

kevinwallimann commented 4 years ago

Currently, extra configuration to the parquet writers needs to be passed like this:

writer.parquet.extra.conf.1=key=value

key=value is split at the = sign, which is very confusing and unexpected. Extra configuration should be added with a prefix writer.parquet.option.key=value

Unfortunately, this is inconsistent with reader.option.key=value, but it's consistent with all other configuration properties which include an identifier for the component. Arguably, reader.option.key=value should be changed to reader.kafka.option.key=value even though this results in properties like reader.kafka.option.kafka.security.protocol

kevinwallimann commented 4 years ago

Don't forget to update the readme with these changes

kevinwallimann commented 4 years ago

Duplicate of #54