search

rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins
https://www.rabbitmq.com/
Other
11.9k stars 3.9k forks source link

Sync AMQP 1.0 Application Properties and AMQP 0.9.1 Properties/Headers in RabbitMQ Shovel #7508

Closed Haigutus closed 7 months ago

Haigutus commented 1 year ago

https://groups.google.com/g/rabbitmq-users/c/2RhTKTYKc7I/m/_dAH1cUqBQAJ

RabbitMQ Shovel currently does not keep the AMQP 1.0 Application Properties and AMQP 0.9.1 Properties/Headers in sync when copying messages between exchanges or queues when transferring messages between AMQP 1.0 and AMQP 0.9.1 brokers.

To improve the functionality of RabbitMQ Shovel, I suggest that the Shovel should always copy the Application Properties and Properties/Headers of messages along with their body, and keep them in sync across all copies of the message. This will ensure that headers can be used for routing when transferring messages between AMQP 1.0 and AMQP 0.9.1 brokers.

michaelklishin commented 1 year ago

Those properties do not have a perfect 1:1 match. At some point we plan to use AMQP 1.0 properties internally for all messages but we are not there yet.

So this is significantly more involved than it may seem.

Haigutus commented 1 year ago

Hi those are user defined key and value pairs, so no need to map anything, just carry over.

It seems that for AMQP 0.9.1 -> AMQP 1.0 it is actually already done, but other way it seems not, but I am not familiar enough with the code base

https://github.com/rabbitmq/rabbitmq-server/blob/942c4ab5426a376f83fff32114b98306b9ddd6f2/deps/rabbitmq_shovel/src/rabbit_amqp10_shovel.erl#L397

Haigutus commented 1 year ago

For reference

AMQP 1.0

Application Properties:

Any application-specific data, encoded as key-value pairs.

AMQP 0.9.1

Properties/Headers: Additional metadata about the message, encoded as key-value pairs.

Haigutus commented 1 year ago

Also it seems, that for AMQP 1.0 plugin this has already been implemented. https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_amqp1_0/README.md image

Do these settings also apply for shovel?

michaelklishin commented 7 months ago

This was addressed (without any configuration settings) in https://github.com/rabbitmq/rabbitmq-server/pull/10037 for 3.12.x.

As of right now, https://github.com/rabbitmq/rabbitmq-server/pull/10022 is still is WIP to see if it can be refactored to benefit from the recent message container improvements.