search

awslabs / aws-glue-schema-registry

AWS Glue Schema Registry Client library provides serializers / de-serializers for applications to integrate with AWS Glue Schema Registry Service. The library currently supports Avro, JSON and Protobuf data formats. See https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html to get started.
Apache License 2.0
126 stars 96 forks source link

Apache Druid client for AWS MSK + AWS glue schema registry @question #95

Open hi4murali opened 2 years ago

hi4murali commented 2 years ago

Hi, Currently we use Apache Druid as Kafka consumer/ producer with Confluent Kafka + Confluent Schema Registry. We are assessing to move to AWS MSK however, not sure if the Apache Druid will continue as Kafka consumer/ producer along with + AWS glue schema registry. Is this supported? If so, is there any reference/ link on how to configure AWS glue schema registry with Apache Druid?

Thanks, Murali

vanessapinto257 commented 2 years ago

Hi,

Thanks for reaching out! Which data format would you use with Glue Schema Registry?

hi4murali commented 2 years ago

Hi, We will need both both Avro and CSV to be supported initially and eventually JSON too. Does you question mean that integration might be supported but might be limited to certain formats?

blacktooth commented 2 years ago

As of now, we don't officially support Apache Druid. If this a blocker for you to use leverage GSR, we will prioritize it next year.

As a workaround, you can still use this library to serialize / de-serialize (Avro, JSON) messages before sending / receiving from Apache Druid if feasible.

Below classes support platform independent serialization / de-serialization capabilities,

GlueSchemaRegistrySerializerImpl

GlueSchemaRegistryDeserializerImpl

Please let us know if this is helpful.

SiHicklingFloow commented 2 months ago

This would be useful - we're using AWS Glue Schema Registry with Avro payloads and would love to use Druid for analytics.