cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.17k stars 3.82k forks source link

changefeedccl: support aws glue schema registry #103084

Open amruss opened 1 year ago

amruss commented 1 year ago

When you create a schema registry using the glue the schema registry, you only get exposed an ARN number, not an https endpoint. It also looks like you need to pass the following parameters in through the producer: properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, GlueSchemaRegistryKafkaSerializer.class.getName()); properties.put(AWSSchemaRegistryConstants.DATA_FORMAT, DataFormat.AVRO.name()); properties.put(AWSSchemaRegistryConstants.AWS_REGION, "us-east-1"); properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, "my-registry"); properties.put(AWSSchemaRegistryConstants.SCHEMA_NAME, "my-schema");

The MSK PM for AWS said they recommend using Glue for schema registry. It seems like our existing schema registry support doesn't work with it. We should support Glue.

https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html

Jira issue: CRDB-27825

Epic CRDB-28877

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/cdc

amruss commented 1 year ago

Seems like we would need to use this library: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/glue