Server crash

[fdmota]

Our Rabbitmq cluster crashes with the bellow error. Could help me to understand why? Context: all our queues are configured with

Features
  x-max-priority:1
   arguments:
     x-message-deduplication:true
     durable:true

Everything works fine for a while until the error happens and the cluster becomes unstable. Also even though the queue has x-message-deduplication:true the message don't always have the x-deduplication-header and the x-cache-size was never configured because according to documentation at queue level it is not necessary.

** Last message in was cleanup
** When Server state == #{}
** Reason for termination ==
** {{case_clause,
        {aborted,
            {badarg,
                [cache_queue__sepolia_calculate_daily_volumes_dead_letter,
                 [{{cache_queue__sepolia_calculate_daily_volumes_dead_letter,
                       '$1','$2'},
                   [{'>',1686324953790,'$2'}],
                   ['$1']}]]}}},
    [{'Elixir.RabbitMQMessageDeduplication.Cache',delete_expired_entries,1,
         [{file,"lib/rabbitmq_message_deduplication/cache.ex"},{line,118}]},
     {'Elixir.Enum','-each/2-lists^foreach/1-0-',2,
         [{file,"lib/enum.ex"},{line,937}]},
     {'Elixir.RabbitMQMessageDeduplication.CacheManager',handle_info,2,
         [{file,"lib/rabbitmq_message_deduplication/cache_manager.ex"},
          {line,138}]},
     {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]},
     {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

  crasher:
    initial call: 'Elixir.RabbitMQMessageDeduplication.CacheManager':init/1
    pid: <0.20899.0>
    registered_name: 'Elixir.RabbitMQMessageDeduplication.CacheManager'
    exception error: no case clause matching {aborted,
                                              {badarg,
                                               [cache_queue__sepolia_calculate_daily_volumes_dead_letter,
                                                [{{cache_queue__sepolia_calculate_daily_volumes_dead_letter,
                                                   '$1','$2'},
                                                  [{'>',1686324953790,'$2'}],
                                                  ['$1']}]]}}
      in function  'Elixir.RabbitMQMessageDeduplication.Cache':delete_expired_entries/1 (lib/rabbitmq_message_deduplication/cache.ex, line 118)
      in call from 'Elixir.Enum':'-each/2-lists^foreach/1-0-'/2 (lib/enum.ex, line 937)
      in call from 'Elixir.RabbitMQMessageDeduplication.CacheManager':handle_info/2 (lib/rabbitmq_message_deduplication/cache_manager.ex, line 138)
      in call from gen_server:try_dispatch/4 (gen_server.erl, line 1123)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1200)
    ancestors: [rabbit_sup,<0.20328.0>]
    message_queue_len: 0
    messages: []
    links: [<0.20448.0>,<0.20385.0>]
    dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,
                                next => #Fun<rand.0.34006561>,type => exsss,
                                uniform => #Fun<rand.1.34006561>,
                                uniform_n => #Fun<rand.2.34006561>},
                              [7327905855751085|54965543541016194]}}]
    trap_exit: false
    status: running
    heap_size: 6772
    stack_size: 28
    reductions: 611647153
  neighbours:

    supervisor: {local,rabbit_sup}
    errorContext: child_terminated
    reason: {{case_clause,
                 {aborted,
                     {badarg,
                         [cache_queue__sepolia_calculate_daily_volumes_dead_letter,
                          [{{cache_queue__sepolia_calculate_daily_volumes_dead_letter,
                                '$1','$2'},
                            [{'>',1686324953790,'$2'}],
                            ['$1']}]]}}},
             [{'Elixir.RabbitMQMessageDeduplication.Cache',
                  delete_expired_entries,1,
                  [{file,"lib/rabbitmq_message_deduplication/cache.ex"},
                   {line,118}]},
              {'Elixir.Enum','-each/2-lists^foreach/1-0-',2,
                  [{file,"lib/enum.ex"},{line,937}]},
              {'Elixir.RabbitMQMessageDeduplication.CacheManager',
                  handle_info,2,
                  [{file,
                       "lib/rabbitmq_message_deduplication/cache_manager.ex"},
                   {line,138}]},
              {gen_server,try_dispatch,4,
                  [{file,"gen_server.erl"},{line,1123}]},
              {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]},
              {proc_lib,init_p_do_apply,3,
                  [{file,"proc_lib.erl"},{line,240}]}]}
    offender: [{pid,<0.20899.0>},
               {id,'Elixir.RabbitMQMessageDeduplication.CacheManager'},
               {mfargs,
                   {'Elixir.RabbitMQMessageDeduplication.CacheManager',
                       start_link,[]}},
               {restart_type,transient},
               {significant,false},
               {shutdown,300000},
               {child_type,worker}]

    supervisor: {local,rabbit_sup}
    errorContext: shutdown
    reason: reached_max_restart_intensity
    offender: [{pid,<0.20899.0>},
               {id,'Elixir.RabbitMQMessageDeduplication.CacheManager'},
               {mfargs,
                   {'Elixir.RabbitMQMessageDeduplication.CacheManager',
                       start_link,[]}},
               {restart_type,transient},
               {significant,false},
               {shutdown,300000},
               {child_type,worker}]```

[ipeleg]

+1 on this issue, could it be some sort of conflict with the delayed message plugin? if not could be wrong there?

[noxdafox]

Hello,

do I understand correctly that this is a de-duplication queue?

• If so, how is the queue configured?
• What functionalities of the queue are you using?
  • I see it's also a priority queue, are you using other functionalities such as:
  • max queue size
  • message TTL

In case of a deduplication queue, it is correct to assume that x-cache-size is not needed. The plugin should also work fine without a header on all messages.