search

noxdafox / rabbitmq-message-deduplication

RabbitMQ Plugin for filtering message duplicates
Mozilla Public License 2.0
275 stars 34 forks source link

Discussion/feature request - configurable queue deduplication semantics #53

Open pbillen opened 4 years ago

pbillen commented 4 years ago

Hi,

I would like to discuss something regarding queue deduplication semantics, in the hope to get some interest for extending this plugin.

One could define two definitions for queue deduplication:

  1. do not enqueue a message if an equivalent message is already present in the queue
  2. do not enqueue a message if an equivalent message is already present and not being processed in the queue

From my understanding in https://github.com/noxdafox/rabbitmq-message-deduplication/issues/51, this plugin currently implements the first definition, being the most intuitive and safe/correct definition.

For the project I am working on, it would be useful if we could support the second definition as well. (Please let me know if you want me to elaborate on this. In short, I am using RabbitMQ to notify workers that tasks have been added to the database, available to be executed. Here, it is only safe to prevent duplicates under the second definition, as we cannot know for sure that an ongoing consumer will see all additional tasks being added during its execution loop.)

Is this something that could be implemented in this plugin? We could define the deduplication definition to be applied via some argument property when creating the queue?

Note: It's important to document that we can end up with "real" duplicates in the queue. For example, assume we are preventing duplicates under the second definition. Assume a certain consumer, with manual acknowledgement enabled, is processing message X while we enqueue another message X. Under the second definition, this is allowed. If now the consumer crashes or rejects message X, it is given back to the queue. At this point, we have two X messages.

Thank you.

pbillen commented 4 years ago

After reading https://github.com/noxdafox/rabbitmq-message-deduplication/issues/27, I believe this is a duplicate (no pun intented) of that issue.