AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

Fix inconsistencies between CommandLineIngestorDriver and PropertiesIngestorDriver #120

Closed kevinwallimann closed 4 years ago

kevinwallimann commented 4 years ago

Fix inconsistencies:

  1. In CommandLineIngestorDriver whitespace in key and value is not trimmed, while it is trimmed in PropertiesIngestorDriver. E.g. key1 = value1 will be stored as key1 (with whitespace at end) with CommandLineIngestorDriver, but as key1 in PropertiesIngestorDriver. The behavior of PropertiesIngestorDriver is the expected one.

  2. An empty property, i.e. some.key= causes an exception in CommandLineIngestorDriver, but is allowed in PropertiesIngestorDriver. This is inconsistent. Either it should always cause an exception, or it should be allowed. Probably the behavior of PropertiesIngestorDriver should be favoured since it directly uses the behaviour of Configuration2. Users familiar with Configuration2 would find a differing behaviour unexpected.