search

awslabs / aws-glue-schema-registry

AWS Glue Schema Registry Client library provides serializers / de-serializers for applications to integrate with AWS Glue Schema Registry Service. The library currently supports Avro, JSON and Protobuf data formats. See https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html to get started.
Apache License 2.0
126 stars 96 forks source link

consume records with glue schema registry on cli #36

Open yaboong opened 3 years ago

yaboong commented 3 years ago

I used to use the command like below

docker exec -it schema-registry /usr/bin/kafka-avro-console-consumer \
--bootstrap-server kafka:29092 \
--property  \
--from-beginning \
--topic some-topic

but when testing with glue schema registry, I got stuck because glue schema registry doesn't provide http endpoint. Do you guys have any plan to support this feature or support http endpoint for glue?

blacktooth commented 3 years ago

Hi,

Glue Schema Registry doesn't have endpoints to be configured because it's a fully managed service. The Schema Registry APIs are accessible using Glue endpoints.

We haven't tested kafka-avro-console-consumer with Glue Schema Registry. We will test it and get back to you.

j0r0 commented 3 years ago

This will be very helpful. One wouldn't need to write a special consumer just to see the data. Not to mention other tools like Kafkatool2...

Confluent platform supports that out-of-the-box

blacktooth commented 3 years ago

For this tool to work, there should be Glue Schema Registry implementation of this interface. The implementation can use the GlueSchemaRegistryDeserializerImpl class to decode the messages. There needs to be a small wrapper written around KafakaConsoleConsumer to consume the records with this newly implemented interface.