AWS Glue Schema Registry Client library provides serializers / de-serializers for applications to integrate with AWS Glue Schema Registry Service. The library currently supports Avro, JSON and Protobuf data formats. See https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html to get started.
But when using this, we are getting the below error
[Worker-002f6cdee8de409ee] [2022-10-18 12:33:12,501] ERROR [New-connect|task-0] WorkerSinkTask{id=New-connect-0} Error converting message value in topic 'NewTopic' partition 0 at offset 0 and timestamp 1666096125363: Converting byte[] to Kafka Connect data failed due to serialization error: (org.apache.kafka.connect.runtime.WorkerSinkTask:547)
org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:
Can someone help with the root cause of this error. from what I understood, there is a difference between the serializer and de serializer. Is there a solution for this?
We are trying to use MSK sink connector for data that is passed in avro format to MSK. The producer uses aws-glue-schema-registry
Configuration for MSK connect is below connector.class=io.confluent.connect.s3.S3SinkConnector format.class=io.confluent.connect.s3.format.avro.AvroFormat flush.size=1 schema.compatibility=BACKWARD tasks.max=2 topics=MSKTutorialTopic storage.class=io.confluent.connect.s3.storage.S3Storage topics.dir=msk-delete s3.bucket.name=clickstream-analytics s3.region=us-east-1 partitioner.class=io.confluent.connect.storage.partitioner.TimeBasedPartitioner path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH partition.duration.ms=600000 locale=en timezone=UTC timestamp.extractor=Record value.converter.schemaAutoRegistrationEnabled=true value.converter.registry.name=datalake-schema-registry value.converter.avroRecordType=GENERIC_RECORD value.converter.region=us-east-1 value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter key.converter=org.apache.kafka.connect.storage.StringConverter
But when using this, we are getting the below error [Worker-002f6cdee8de409ee] [2022-10-18 12:33:12,501] ERROR [New-connect|task-0] WorkerSinkTask{id=New-connect-0} Error converting message value in topic 'NewTopic' partition 0 at offset 0 and timestamp 1666096125363: Converting byte[] to Kafka Connect data failed due to serialization error: (org.apache.kafka.connect.runtime.WorkerSinkTask:547) org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:
Can someone help with the root cause of this error. from what I understood, there is a difference between the serializer and de serializer. Is there a solution for this?