noxdafox / rabbitmq-message-deduplication

RabbitMQ Plugin for filtering message duplicates
Mozilla Public License 2.0
271 stars 33 forks source link

Queue crashes when is created with x-message-deduplication=true parameter on RabbitMQ 3.13.0, Plugin version 0.6.2 #108

Closed dabogee closed 2 weeks ago

dabogee commented 3 months ago

Steps:

PS: Code works on RabbitMQ 3.11.0 (as mentioned on release page for 0.6.2 ) then I have to downgrade RabbitMQ to get it works.

Actual: There is error message in the logs

2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>     supervisor: {<0.64723.0>,rabbit_amqqueue_sup}
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>     errorContext: child_terminated
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>     reason: {function_clause,
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                 [{'Elixir.RabbitMQMessageDeduplication.Queue','duplicate?',
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                      [{amqqueue,
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           {resource,<<"dev">>,queue,<<"test-queue">>},
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           true,false,none,
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           [{<<"x-message-deduplication">>,bool,true},
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                            {<<"x-cache-size">>,long,10000}],
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           <0.64739.0>,[],[],[],undefined,undefined,[],[],
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           crashed,0,[],<<"dev">>,
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           #{user => <<"admin">>},
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                           rabbit_classic_queue,#{}},
2024-04-01 12:58:33.267838+00:00 [error] <0.64723.0>                       {mc,mc_amqpl,
...

And then receive a popup message on UI: "rejected Unable to publish message. Check queue limits."

noxdafox commented 3 months ago

Hello,

the error is incomplete, could you please paste the whole error message?

Nuranto commented 2 months ago

I confirm this issue.

supervisor: {<0.666.0>,rabbit_amqqueue_sup}

errorContext: child_terminated

reason: {function_clause,

[{'Elixir.RabbitMQMessageDeduplication.Queue','duplicate?',

[{amqqueue,

{resource,<<"/">>,queue,

<<"queue_name">>},

true,false,none,

[{<<"x-message-deduplication">>,bool,true}],

<0.1771.0>,[],[],[],undefined,undefined,[],[],

crashed,0,[],<<"/">>,

#{user => <<"####">>},

rabbit_classic_queue,#{}},

{mc,mc_amqpl,

{content,60,none,

<<####>>,

rabbit_framing_amqp_0_9_1,

[<<"{\"message\":\"####\"}">>]},

#{id =>

<<####>>,

x => <<"####">>,

rk => [<<"queue_name">>],

rts => 1714046802630}}],

[{file,

"lib/rabbitmq_message_deduplication/rabbit_message_deduplication_queue.ex"},

{line,524}]},

{'Elixir.RabbitMQMessageDeduplication.Queue',is_duplicate,2,

[{file,

"lib/rabbitmq_message_deduplication/rabbit_message_deduplication_queue.ex"},

{line,431}]},

{rabbit_amqqueue_process,maybe_deliver_or_enqueue,3,

[{file,"rabbit_amqqueue_process.erl"},{line,728}]},

{rabbit_amqqueue_process,handle_cast,2,

[{file,"rabbit_amqqueue_process.erl"},{line,1576}]},

{gen_server2,handle_msg,2,

[{file,"gen_server2.erl"},{line,1056}]},

{proc_lib,init_p_do_apply,3,

[{file,"proc_lib.erl"},{line,241}]}]}

offender: [{pid,<0.1771.0>},

{id,rabbit_amqqueue},

{mfargs,

{rabbit_prequeue,start_link,

[{amqqueue,

{resource,<<"/">>,queue,

<<"queue_name">>},

true,false,none,

[{<<"x-message-deduplication">>,bool,true}],

<0.3998.0>,[],[],[],undefined,undefined,[],[],

stopped,0,[],<<"/">>,

#{user => <<"####">>},

rabbit_classic_queue,#{}},

recovery,<0.665.0>]}},

{restart_type,transient},

{significant,true},

{shutdown,600000},

{child_type,worker}]

 [error] <0.666.0>
heikkimu commented 2 months ago

Hey hey, same problem here with RabbitMQ 3.13.2.

phil16727 commented 2 months ago

Same Issue here!!

gomoripeti commented 1 month ago

The problem is most likely the introduction of "message containers" (different internal representation of messages) in RabbitMQ 3.13.0. I'm looking at fixing this.

noxdafox commented 2 weeks ago

Issue should be resolved in release 0.6.3. Re-open this ticket if the problem persists.