Currently, the ConfluentAvroDecodingTransformer removes any columns other than key and value by only selecting these.
However, for #177, the offset and partition from the original dataframe from the KafkaStreamReader need to be retained across the ConfluentAvroDecodingTransformer such that the source offset and partition are written to the sink topic for each message.
Feature
An option should be added to specify columns that should be kept across the decoding. If there is a column name collision, an exception should be thrown. Column name collisions may be solved by inserting an instance of the ColumnRenamingStreamTransformer in front of it in the pipeline
Background
Currently, the
ConfluentAvroDecodingTransformerremoves any columns other thankeyandvalueby only selecting these.However, for #177, the offset and partition from the original dataframe from the
KafkaStreamReaderneed to be retained across theConfluentAvroDecodingTransformersuch that the source offset and partition are written to the sink topic for each message.Feature
An option should be added to specify columns that should be kept across the decoding. If there is a column name collision, an exception should be thrown. Column name collisions may be solved by inserting an instance of the
ColumnRenamingStreamTransformerin front of it in the pipeline