Boolean config parameters should be interpreted consistently - Githubissues

AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark

Apache License 2.0

44 stars 13 forks source link

Boolean config parameters should be interpreted consistently #165

Closed kevinwallimann closed 3 years ago

kevinwallimann commented 4 years ago

Currently, boolean config parameters, such as

transformer.{transformer-id}.consume.keys in ConfluentAvroDecodingTransformer
transformer.{transformer-id}.produce.keys in ConfluentAvroEncodingTransformer
writer.parquet.metadata.check
reader.parquet.wait.for.files

don't map config values to booleans in a consistent way.

The mapping should be done in the following way:

Config property	Option[Boolean]
Key present and value is empty	throw exception
Key not present	`None`
"true"	`Some(true)`
"True"	`Some(true)`
"false"	`Some(false)`
"False"	`Some(false)`
Any other value, such as "0", "1", etc.	throw exception