search

AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

Express ConfluentAvroKafkaDecoder as transformer #128

Closed kevinwallimann closed 4 years ago

kevinwallimann commented 4 years ago

KafkaStreamReader.read should return DataFrame ConfluentAvroKafkaDecoder.decode could then accept DataFrame and return DataFrame, making it a Transformer, e.g. za.co.absa.hyperdrive.ingestor.implementation.transformer.avro.ConfluentAvroDecoderTransformer

Then, the Decoder component is not needed anymore.

How to migrate Hyperdrive-Trigger

  1. Replace 
    "component.decoder=za.co.absa.hyperdrive.ingestor.implementation.decoder.avro.confluent.ConfluentAvroKafkaStreamDecoder"

    with

    "component.transformer.id.0=confluent.avro.decoder", "component.transformer.class.confluent.avro.decoder=za.co.absa.hyperdrive.ingestor.implementation.transformer.avro.ConfluentAvroDecoderTransformer"
  2. Replace 
    decoder.avro

    with

    transformer.confluent.avro.decoder