Particular / NServiceBus.AzureStorageQueues

Azure Storage Queues Transport for NServiceBus
https://docs.particular.net/nservicebus/azure-storage-queues/
Other
4 stars 3 forks source link

Native Integrations are not possible with headerless transports #844

Open kbaley opened 2 years ago

kbaley commented 2 years ago

This issue was transferred from a private repository

Native integrations (aka Third Party Integrations) allow NServiceBus endpoints to receive messages from external systems.

In order for native integrations to work, NServiceBus requires a mandatory header to discover the enclosed message type. While this works great with the transports that use underlying services supporting headers, native integration is impossible with transports that have no support for native headers. I.e. AzureStorageQueues

What happens a message w/o headers arrives?

A message would be wrapped in a transport-specific envelope that is expected to be found. In case of native integrations, if the source system emits messages as-is, NServiceBus message wrapper will be absent. The transport will fail to understand the message and it will be considered as poisonous and moved to the error queue.

Why does it matter?

An example would be a system that would want to process events from Azure EventGrid. An NServiceBus endpoint could be specified as a destination for EventGrid messages (subscriber) and EventGrid would automatically push messages to the endpoints' input queue.

EventGrid messages have a very specific schema and no headers. This means ASQ transport will fail to ingest those messages at the pump level and move them to the error queue.

A workaround would be to introduce a "man in the middle" to

This adds unnecessary complexity and additional failing points to native integrations.

What are options?

NServiceBus Core could assist in standardization of native integrations by having a contract/seam of how transports interpret incoming messages. There would be a default behaviour, allowing transports to override it. Transports such as ASQ/SQS would be able to understand native message formats (EventGrid, CloudEvents, etc) to read those and deserialize properly.

SeanFeldman commented 2 years ago

The statement below is inaccurate/wrong:

A workaround would be to introduce a "man in the middle" to

  • Receive EventGrid messages on a separate queue/process
  • Wrap those for ASQ transport to understand those
  • Push wrapped messages to the NServiceBus endpoint

This adds unnecessary complexity and additional failing points to native integrations.

There's no need to receive on a separate queue and push it later to NSB endpoint. ASQ has a feature to deal with 3rd party messages that do not comply with NSB's MessageWrapper, called Custom Envelope Wrapper.

mauroservienti commented 11 months ago

This has been solved for SQS in https://github.com/Particular/NServiceBus.AmazonSQS/pull/2341