search

awslabs / aws-glue-schema-registry

AWS Glue Schema Registry Client library provides serializers / de-serializers for applications to integrate with AWS Glue Schema Registry Service. The library currently supports Avro, JSON and Protobuf data formats. See https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html to get started.
Apache License 2.0
129 stars 96 forks source link

Error while using MSK s3 sink connector #227

Closed shalinibnair closed 1 year ago

shalinibnair commented 2 years ago

We are trying to use MSK sink connector for data that is passed in avro format to MSK. The producer uses aws-glue-schema-registry

Configuration for MSK connect is below connector.class=io.confluent.connect.s3.S3SinkConnector format.class=io.confluent.connect.s3.format.avro.AvroFormat flush.size=1 schema.compatibility=BACKWARD tasks.max=2 topics=MSKTutorialTopic storage.class=io.confluent.connect.s3.storage.S3Storage topics.dir=msk-delete s3.bucket.name=clickstream-analytics s3.region=us-east-1 partitioner.class=io.confluent.connect.storage.partitioner.TimeBasedPartitioner path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH partition.duration.ms=600000 locale=en timezone=UTC timestamp.extractor=Record value.converter.schemaAutoRegistrationEnabled=true value.converter.registry.name=datalake-schema-registry value.converter.avroRecordType=GENERIC_RECORD value.converter.region=us-east-1 value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter key.converter=org.apache.kafka.connect.storage.StringConverter

But when using this, we are getting the below error [Worker-002f6cdee8de409ee] [2022-10-18 12:33:12,501] ERROR [New-connect|task-0] WorkerSinkTask{id=New-connect-0} Error converting message value in topic 'NewTopic' partition 0 at offset 0 and timestamp 1666096125363: Converting byte[] to Kafka Connect data failed due to serialization error: (org.apache.kafka.connect.runtime.WorkerSinkTask:547) org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:

Can someone help with the root cause of this error. from what I understood, there is a difference between the serializer and de serializer. Is there a solution for this?

mohitpali commented 1 year ago

Could you provide us with more logs and information to understand it better ?

blacktooth commented 1 year ago

Please feel free to re-open this issue if you need further help with this investigation.