KC with Event Hubs Sink 1.0.0

[egonbaren]

Hi. I want to use Camel Sink KafkaConnector (1.0.0) with Azure Event Hubs forwarding event to Azure, but it doesn't look like to forward headers. There is possibility to remove headers explicitly. Should default behaviour forward all the headers or not?

camel.remove.headers.pattern=CamelAwsS3BucketName https://camel.apache.org/camel-kafka-connector/next/user-guide/remove-headers.html

Events are forwarder but headers are missing, are there any plans to enable it?

[oscerd]

I think there is a problem in the component from Plain camel. I'll check it

[oscerd]

Also what headers are you talking about?

[johnnyreilly]

Thanks @oscerd - a number of us have been working on this this week and encountered surprising behaviour where all headers were being stripped. I'll see if I can get one of the team to better express the issue than I can. Thanks for responding!

[oscerd]

I'll have a look next week. I guess there is something in the component but need to double check. If you have example of record coming from Kafka, please post it

[johnnyreilly]

Thanks @oscerd - will see what we can do!

[Doug-North]

Here is an example of a message + headers, formatted with kcat:

Key (-1 bytes):
  Value (141 bytes): {"CorrelationId":"a529A3D7-e15f-3d21e-315t6r5-e54csda23dbadQwfh3a1","Message":"Example","Timestamp":"2022-01-22T17:19:43.459939+00:00"}
  Headers: 
  Digest=U0hBLTUxMi8yNTY9dTkAocasnkjzvASOfuqnfsoiuds87101jsmNiTGQ2YUdFWT0=,
  Signature-Input=bW9uZXNNcaiuhasfhASIUWqosidfachsuczkjxvnduaW5nLWtleS0xIg==,
  Signature=bW9uZAkjanscavbchjzxbchbauysfwqoaicocxjaoDRvRFFyeXhURDJJklghyu6==

[oscerd]

Thanks. I'll take a look next week

[oscerd]

At first sight it seems we aren't taking care of the headers in the component producer.

https://github.com/apache/camel/blob/camel-3.14.0/components/camel-azure/camel-azure-eventhubs/src/main/java/org/apache/camel/component/azure/eventhubs/operations/EventHubsProducerOperations.java#L140

This is something we need to double check. But, there is no mapping between Camel headers and EventData object properties field.

[oscerd]

https://issues.apache.org/jira/browse/CAMEL-17543

[oscerd]

To make them pass you should prepend "CamelHeader." to your header name. In that way you should see them routing, but the problem on the component remains.

[valdar]

Here is an example of a message + headers, formatted with kcat:

Key (-1 bytes):
  Value (141 bytes): {"CorrelationId":"a529A3D7-e15f-3d21e-315t6r5-e54csda23dbadQwfh3a1","Message":"Example","Timestamp":"2022-01-22T17:19:43.459939+00:00"}
  Headers: 
  Digest=U0hBLTUxMi8yNTY9dTkAocasnkjzvASOfuqnfsoiuds87101jsmNiTGQ2YUdFWT0=,
  Signature-Input=bW9uZXNNcaiuhasfhASIUWqosidfachsuczkjxvnduaW5nLWtleS0xIg==,
  Signature=bW9uZAkjanscavbchjzxbchbauysfwqoaicocxjaoDRvRFFyeXhURDJJklghyu6==

@Doug-North @johnnyreilly so if the above is an example of message sent to the sink connector those headers should be called: CamelHeader.Digest, CamelHeader.Signature-Input and CamelHeader.Signature.

Would be nice if you could try with those names and see what happens.

[oscerd]

The component has been fixed by the way.