search

AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

Extract encoding part of KafkaStreamWriter to transformer component #129

Closed kevinwallimann closed 4 years ago

kevinwallimann commented 4 years ago

KafkaStreamWriter should not be dependent on Abris (confluent and Avro) Thus, the encoding part should be extracted to a preceding transformer component

A new transformer should be created, e.g. za.co.absa.hyperdrive.ingestor.implementation.transformer.avro.ConfluentAvroEncoderTransformer

Breaking changes Configuration properties will need to be adjusted

How to migrate Hyperdrive-Trigger

  1. Replace 

    "component.writer

    with

    "component.transformer.id.2=confluent.avro.encoder", "component.transformer.class.confluent.avro.encoder=za.co.absa.hyperdrive.ingestor.implementation.transformer.avro.ConfluentAvroEncoderTransformer",
"component.writer

  2. Replace

    writer.kafka.schema

    with

    transformer.confluent.avro.encoder.schema

  3. Replace

    writer.kafka.value

    with

    transformer.confluent.avro.encoder.value

  4. Replace

    writer.kafka.produce.keys

    with

    transformer.confluent.avro.encoder.produce.keys

  5. Replace

    writer.kafka.key

    with

    transformer.confluent.avro.encoder.key

  6. Replace

    writer.kafka.option

    with

    transformer.confluent.avro.encoder.option