search

apache / camel-kafka-connector

Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors
https://camel.apache.org
Apache License 2.0
154 stars 102 forks source link

KafkaConnect Camel RabbitMQ integration with AWS Glue #1565

Open davido912 opened 1 year ago

davido912 commented 1 year ago

Hey all. We're currently using schemas stored in AWS Glue and have tried to use the RabbitMQ Camel KafkaConnector. Seems to work pretty good but when trying to configure it with Avro value converter, as such:

    value.converter.avroRecordType: GENERIC_RECORD
    value.converter.schemaName: banking.ledger.SubLedgerCreatedEventMessage
    value.converter.registry.name: streaming-registry
    value.converter: com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
    value.converter.region: eu-central-1

We're getting the following error:

org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: org.apache.kafka.connect.data.Struct to the required type: byte[] with value Struct{subLedger=Struct{...

The values in the struct seem to be the deserialized value. Is there any specific configuration I'm required to add?

Would appreciate any help, thanks! (or even knowing if this is not supported)

oscerd commented 2 months ago

There is no experimentation with AWS Glue at this stage. I'll leave this open if someone tried that.