search

Azure / kafka-sink-azure-kusto

Kafka sink for Kusto
MIT License
48 stars 24 forks source link

Adding support for Avro and JSON formats with schema registry #31

Closed fahadhasher closed 4 years ago

fahadhasher commented 4 years ago

Added support for Avro and JSON with schema registry.

Using AvroConverter to consume records from Kafka Topic: io.confluent.connect.avro.AvroConverter

  1. Connector will handle avro records as Struct and ingest records to KustoDB table with mapping as avro.
  2. Connector will handle json-with-schema records as Struct and ingest records to KustoDB table with mapping as json.

Using JsonConverter to consume records from Kafka Topic: org.apache.kafka.connect.json.JsonConverter

  1. Connector will handle records as Map and ingest records to KustoDB table with mapping as json.

Removed eventDataCompression from implementation. Updated dlq config to misc.deadletterqueue. Updated documentation and configuration file.

fahadhasher commented 4 years ago

@ohadbitt Please review this PR instead of PR#30. Thanks!

fahadhasher commented 4 years ago

@ohadbitt we have resolved all the comments, please take a look. Thanks!