search

noxdafox / rabbitmq-message-deduplication

RabbitMQ Plugin for filtering message duplicates
Mozilla Public License 2.0
271 stars 33 forks source link

Can key be removed from cache, after BasicAck and some time has ellapsed? #89

Open torgashov opened 1 year ago

torgashov commented 1 year ago

Hello! Is it possible to implement the deletion of keys from the cache, messages that have been removed from the queue (BasicAck) after some time has elapsed? If i right, keys stored in cache after message removed from queue (BasicAck).

noxdafox commented 1 year ago

Hello,

I am not sure I understand what is the use case here. What is you are trying to achieve?

If you want to deduplicate based on elapsed time, you can use exchange-level deduplication.

AnirudhSA commented 1 year ago

Hi @noxdafox ,

@torgashov If I'm not wrong, Probably means that is it possible to remove the key from cache after the message has been acknowledged and removed from the queue. Is it possible to have a parameter at exchange level where this auto-deletion of key can be enabled (which deletes the key after the message is removed from the queue)?

noxdafox commented 1 year ago

I know what the OP is asking.

What is being asked is complicated both from a development and operation point of view.

The first question I have in mind is: based on what it is decided for how long to wait? What if a message arrives right after that time? If time is what matter, why not using the exchange level deduplication which allows a predictable deduplication window?

If a queue is empty, there will not be duplicate messages. Yet the same message can arrive over and over again. If a queue is full, there is no way to predict for how long a message will be deduplicated. In the worst case scenario (no consumer) a message might be deduplicated forever.

Hence, I would like to understand from a high level what is the overall design of the application. A design which, IMHO, leads to overly complicated and unpredictable behaviour.