search

adobe / experience-platform-streaming-connect

Experience platform Streaming Connectors for Data Ingress
Apache License 2.0
23 stars 20 forks source link

Add support for ingesting AVRO format into Kafka Sink Connector #49

Closed coolaries30 closed 8 months ago

coolaries30 commented 1 year ago

Subject of the issue

Please add support for ingesting AVRO File Format into AEP.

Your environment

Steps to reproduce

Tell us how to reproduce this issue.

Expected behaviour

AEP is currently unable to ingest AVRO data format for Streaming Kafka Sink Connector.

Actual behaviour

Kafka Sink Connector in AEP should be able to ingest events in AVRO format.

OneCricketeer commented 1 year ago

I assume you mean Confluent encoded records, from their Schema Registry rather than raw Avro encoding? And not "files" as these are binary records?

coolaries30 commented 1 year ago

Hi Jordan,

I mean raw streaming event records in AVRO Format which have an AVRO Schema defined in the Schema Registry. AEP currently only accepts JSON records but we would like to have support for AVRO format as well.

Thanks, Sudhir

On Thu, Apr 27, 2023 at 10:47 PM Jordan Moore @.***> wrote:

I assume you mean Confluent encoded records, from their Schema Registry rather than raw Avro encoding? And not "files" as these are binary records?

— Reply to this email directly, view it on GitHub https://github.com/adobe/experience-platform-streaming-connect/issues/49#issuecomment-1527021320, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVC7LP4HEIXSR6UPASZRZFDXDNKYRANCNFSM6AAAAAAXLU26HE . You are receiving this because you authored the thread.Message ID: @.*** com>

OneCricketeer commented 1 year ago

My question is more about, have you tried this already? What errors are you seeing? 

value.converter=io.confluent...AvroConverter
value.converter.schema.registry.url=...

Or this one if not using any registry?https://github.com/farmdawgnation/registryless-avro-converter

More specifically, AEP exposes a well documented API, separately from this connector. It needs JSON payloads, but this Connector simply deserializes Kafka events defined by the Connect API SinkRecord type and passes them through, and those could be any format that has an associated Converter.

OneCricketeer commented 1 year ago

Non-JSON Converter support should be possible with #54

OneCricketeer commented 9 months ago

The above PR has been merged and docs are added to the README