search

noxdafox / rabbitmq-message-deduplication

RabbitMQ Plugin for filtering message duplicates
Mozilla Public License 2.0
282 stars 34 forks source link

docker hub rabbitmq:3.7.8 with 0.3.4-erl-20 doesn't dedup queue #20

Closed zamnuts closed 6 years ago

zamnuts commented 6 years ago

Using the docker hub version of rabbitmq doesn't perform queue deduplication.

Steps to reproduce...

Dockerfile (make sure you have the ez files next to this file):

FROM rabbitmq:3.7.8-management-alpine
ADD \
  rabbitmq_message_deduplication-0.3.4-erl-20.ez \
  elixir-1.7.2-erl-20.ez \
  /opt/rabbitmq/plugins/

Build with: docker build --tag rmq:3.7.8-with-dedup .

Run the built image: docker run -d -p '15672:15672' rmq:3.7.8-with-dedup (this exposes the management port should you want it).

Run the following commands on the container, this is easier by exec'ing right into the running container via docker exec -it <container id> /bin/bash. The following is verbose to show output of various steps.

rabbitmq-plugins list:

 Configured: E = explicitly enabled; e = implicitly enabled
 | Status: * = running on rabbit@f581a037d713
 |/
[  ] rabbitmq_amqp1_0                  3.7.8
[  ] rabbitmq_auth_backend_cache       3.7.8
[  ] rabbitmq_auth_backend_http        3.7.8
[  ] rabbitmq_auth_backend_ldap        3.7.8
[  ] rabbitmq_auth_mechanism_ssl       3.7.8
[  ] rabbitmq_consistent_hash_exchange 3.7.8
[  ] rabbitmq_event_exchange           3.7.8
[  ] rabbitmq_federation               3.7.8
[  ] rabbitmq_federation_management    3.7.8
[  ] rabbitmq_jms_topic_exchange       3.7.8
[E*] rabbitmq_management               3.7.8
[e*] rabbitmq_management_agent         3.7.8
[  ] rabbitmq_message_deduplication    0.3.4
[  ] rabbitmq_mqtt                     3.7.8
[  ] rabbitmq_peer_discovery_aws       3.7.8
[  ] rabbitmq_peer_discovery_common    3.7.8
[  ] rabbitmq_peer_discovery_consul    3.7.8
[  ] rabbitmq_peer_discovery_etcd      3.7.8
[  ] rabbitmq_peer_discovery_k8s       3.7.8
[  ] rabbitmq_random_exchange          3.7.8
[  ] rabbitmq_recent_history_exchange  3.7.8
[  ] rabbitmq_sharding                 3.7.8
[  ] rabbitmq_shovel                   3.7.8
[  ] rabbitmq_shovel_management        3.7.8
[  ] rabbitmq_stomp                    3.7.8
[  ] rabbitmq_top                      3.7.8
[  ] rabbitmq_tracing                  3.7.8
[  ] rabbitmq_trust_store              3.7.8
[e*] rabbitmq_web_dispatch             3.7.8
[  ] rabbitmq_web_mqtt                 3.7.8
[  ] rabbitmq_web_mqtt_examples        3.7.8
[  ] rabbitmq_web_stomp                3.7.8
[  ] rabbitmq_web_stomp_examples       3.7.8

rabbitmq-plugins enable rabbitmq_message_deduplication:

The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_message_deduplication
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@f581a037d713...
WARNING: module 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue' not found, so not scanned for boot steps.
The following plugins have been enabled:
  rabbitmq_message_deduplication

started 1 plugins.

Notice the following logs from the container docker container logs <container id>:

[info] <0.33.0> Application elixir started on node rabbit@2e85580642a5
[info] <0.33.0> Application rabbitmq_message_deduplication started on node rabbit@2e85580642a5
[error] <0.38.0> Loading of /var/lib/rabbitmq/mnesia/rabbit@2e85580642a5-plugins-expand/rabbitmq_message_deduplication-0.3.4/ebin/Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue.beam failed: badfile
[error] emulator beam/beam_load.c(1863): Error loading module 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':
  This BEAM file was compiled for a later version of the run-time system than 20.
  To fix this, please recompile this module with an 20 compiler.
  (Use of opcode 163; this emulator supports only up to 159.)

rabbitmq-plugins list shows the plugin is enabled:

 Configured: E = explicitly enabled; e = implicitly enabled
 | Status: * = running on rabbit@f581a037d713
 |/
[  ] rabbitmq_amqp1_0                  3.7.8
[  ] rabbitmq_auth_backend_cache       3.7.8
[  ] rabbitmq_auth_backend_http        3.7.8
[  ] rabbitmq_auth_backend_ldap        3.7.8
[  ] rabbitmq_auth_mechanism_ssl       3.7.8
[  ] rabbitmq_consistent_hash_exchange 3.7.8
[  ] rabbitmq_event_exchange           3.7.8
[  ] rabbitmq_federation               3.7.8
[  ] rabbitmq_federation_management    3.7.8
[  ] rabbitmq_jms_topic_exchange       3.7.8
[E*] rabbitmq_management               3.7.8
[e*] rabbitmq_management_agent         3.7.8
[E*] rabbitmq_message_deduplication    0.3.4
[  ] rabbitmq_mqtt                     3.7.8
[  ] rabbitmq_peer_discovery_aws       3.7.8
[  ] rabbitmq_peer_discovery_common    3.7.8
[  ] rabbitmq_peer_discovery_consul    3.7.8
[  ] rabbitmq_peer_discovery_etcd      3.7.8
[  ] rabbitmq_peer_discovery_k8s       3.7.8
[  ] rabbitmq_random_exchange          3.7.8
[  ] rabbitmq_recent_history_exchange  3.7.8
[  ] rabbitmq_sharding                 3.7.8
[  ] rabbitmq_shovel                   3.7.8
[  ] rabbitmq_shovel_management        3.7.8
[  ] rabbitmq_stomp                    3.7.8
[  ] rabbitmq_top                      3.7.8
[  ] rabbitmq_tracing                  3.7.8
[  ] rabbitmq_trust_store              3.7.8
[e*] rabbitmq_web_dispatch             3.7.8
[  ] rabbitmq_web_mqtt                 3.7.8
[  ] rabbitmq_web_mqtt_examples        3.7.8
[  ] rabbitmq_web_stomp                3.7.8
[  ] rabbitmq_web_stomp_examples       3.7.8

Now let's test the dedup since rabbitmq is reporting the plugin enabled (despite the beam error):

rabbitmqadmin -u guest -p guest declare queue name=deduptest0 'arguments={"x-message-deduplication": true}'
rabbitmqadmin -u guest -p guest -f long -d 3 list queues

list queues gives (truncated):

--------------------------------------------------------------------------------

                                    vhost: /
                                     name: deduptest0
        arguments.x-message-deduplication: True
                              auto_delete: False
                                        ...
                                  durable: True

Finally, test the dedup:

rabbitmqadmin -u guest -p guest publish routing_key=deduptest0 payload=helloworld0  'properties={"headers":{"x-deduplication-header": "dedup-this"}}'
rabbitmqadmin -u guest -p guest publish routing_key=deduptest0 payload=helloworld1  'properties={"headers":{"x-deduplication-header": "dedup-this"}}'
rabbitmqadmin -u guest -p guest get queue=deduptest0 ackmode=ack_requeue_true count=10 --depth=4

get shows 2 messages:

+-------------+----------+---------------+-------------+---------------+------------------+-------------------------------------------+-------------+
| routing_key | exchange | message_count |   payload   | payload_bytes | payload_encoding | properties.headers.x-deduplication-header | redelivered |
+-------------+----------+---------------+-------------+---------------+------------------+-------------------------------------------+-------------+
| deduptest0  |          | 1             | helloworld0 | 11            | string           | dedup-this                                | False       |
| deduptest0  |          | 0             | helloworld1 | 11            | string           | dedup-this                                | False       |
+-------------+----------+---------------+-------------+---------------+------------------+-------------------------------------------+-------------+

I can also reproduce this using the management web UI.

I assume the plugin isn't loading properly (opcode mismatch error), but rabbitmq is reporting it as loaded anyway. I'm using the plugin built for erlang 20 which rabbitmq is using erlang 20. Please advise.

noxdafox commented 6 years ago

Hello,

thanks for reporting this issue with such details.

The culprit is here:

[error] <0.38.0> Loading of /var/lib/rabbitmq/mnesia/rabbit@2e85580642a5-plugins-expand/rabbitmq_message_deduplication-0.3.4/ebin/Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue.beam failed: badfile
[error] emulator beam/beam_load.c(1863): Error loading module 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':
  This BEAM file was compiled for a later version of the run-time system than 20.
  To fix this, please recompile this module with an 20 compiler.
  (Use of opcode 163; this emulator supports only up to 159.)

During the last release, I might have shuffled the versions by mistake. I just re-compiled the plugin with Erlang-20 and uploaded it. Can you please try with the new binary?

khanhpv commented 6 years ago

Hi,

I removed the old files, and just downloaded elixir 1.7.3. But when I enabled the plugin, it outputted the error below:

root@web-misc-rbmq139:/usr/lib/rabbitmq/lib/rabbitmq_server-3.7.8/plugins# rabbitmq-plugins enable rabbitmq_message_deduplication The following plugins have been configured: rabbitmq_management rabbitmq_management_agent rabbitmq_message_deduplication rabbitmq_web_dispatch Applying plugin configuration to rabbit@web-misc-rbmq139... ** (CaseClauseError) no case clause matching: {:badrpc, {:EXIT, {{:case_clause, :error}, [{:rabbit_boot_steps, :"-run_step/2-lc$^1/1-1-", 1, [file: 'src/rabbit_boot_steps.erl', line: 49]}, {:rabbit_boot_steps, :run_step, 2, [file: 'src/rabbit_boot_steps.erl', line: 49]}, {:rabbit_boot_steps, :"-run_boot_steps/1-lc$^0/1-0-", 1, [file: 'src/rabbit_boot_steps.erl', line: 26]}, {:rabbit_boot_steps, :run_boot_steps, 1, [file: 'src/rabbit_boot_steps.erl', line: 26]}, {:rabbit, :start_apps, 2, [file: 'src/rabbit.erl', line: 554]}, {:rabbit_plugins, :ensure1, 1, [file: 'src/rabbit_plugins.erl', line: 65]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}}} (rabbitmqctl) lib/rabbitmq/cli/plugins/plugins_helpers.ex:175: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/2 (rabbitmqctl) lib/rabbitmq/cli/plugins/plugins_helpers.ex:97: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/4 (rabbitmqctl) lib/rabbitmq/cli/plugins/commands/enable_command.ex:97: anonymous fn/6 in RabbitMQ.CLI.Plugins.Commands.EnableCommand.do_run/2 (elixir) lib/stream.ex:1282: anonymous fn/2 in Stream.iterate/2 (elixir) lib/stream.ex:1466: Stream.do_unfold/4 (elixir) lib/stream.ex:1536: Enumerable.Stream.do_each/4 (elixir) lib/stream.ex:923: Stream.do_enum_transform/7 (elixir) lib/stream.ex:1536: Enumerable.Stream.do_each/4

============= root@web-misc-rbmq139:/usr/lib/rabbitmq/lib/rabbitmq_server-3.7.8/plugins# ls -al total 8816 drwxr-xr-x 1 root root 2704 Oct 16 15:30 . drwxr-xr-x 1 root root 66 Sep 21 15:20 .. -rw-r--r-- 1 root root 107381 Sep 20 22:35 amqp10_client-3.7.8.ez -rw-r--r-- 1 root root 44256 Sep 20 22:35 amqp10_common-3.7.8.ez -rw-r--r-- 1 root root 411361 Sep 20 22:35 amqp_client-3.7.8.ez -rw-r--r-- 1 root root 257564 Sep 20 22:35 cowboy-2.2.2.ez -rw-r--r-- 1 root root 439935 Sep 20 22:35 cowlib-2.1.0.ez -rw-r--r-- 1 root root 3116863 Oct 16 13:06 elixir-1.7.3-erl-20.ez -rw-r--r-- 1 root root 69272 Sep 20 22:35 goldrush-0.1.9.ez -rw-r--r-- 1 root root 101536 Sep 20 22:35 jsx-2.8.2.ez -rw-r--r-- 1 root root 230113 Sep 20 22:35 lager-3.6.3.ez -rw-r--r-- 1 root root 705365 Sep 20 22:35 rabbit_common-3.7.8.ez -rw-r--r-- 1 root root 256296 Sep 20 22:35 rabbitmq_amqp1_0-3.7.8.ez -rw-r--r-- 1 root root 31499 Sep 20 22:35 rabbitmq_auth_backend_cache-3.7.8.ez -rw-r--r-- 1 root root 14414 Sep 20 22:35 rabbitmq_auth_backend_http-3.7.8.ez -rw-r--r-- 1 root root 49360 Sep 20 22:35 rabbitmq_auth_backend_ldap-3.7.8.ez -rw-r--r-- 1 root root 20120 Sep 20 22:35 rabbitmq_auth_mechanism_ssl-3.7.8.ez -rw-r--r-- 1 root root 67938 Sep 20 22:35 rabbitmq_aws-3.7.8.ez -rw-r--r-- 1 root root 23674 Sep 20 22:35 rabbitmq_consistent_hash_exchange-3.7.8.ez -rw-r--r-- 1 root root 16066 Sep 20 22:35 rabbitmq_event_exchange-3.7.8.ez -rw-r--r-- 1 root root 238029 Sep 20 22:35 rabbitmq_federation-3.7.8.ez -rw-r--r-- 1 root root 14431 Sep 20 22:35 rabbitmq_federation_management-3.7.8.ez -rw-r--r-- 1 root root 27032 Sep 20 22:35 rabbitmq_jms_topic_exchange-3.7.8.ez -rw-r--r-- 1 root root 1037386 Sep 20 22:35 rabbitmq_management-3.7.8.ez -rw-r--r-- 1 root root 179342 Sep 20 22:35 rabbitmq_management_agent-3.7.8.ez -rw-r--r-- 1 root root 43816 Oct 16 13:06 rabbitmq_message_deduplication-0.3.4-erl-20.ez -rw-r--r-- 1 root root 156261 Sep 20 22:35 rabbitmq_mqtt-3.7.8.ez -rw-r--r-- 1 root root 25829 Sep 20 22:35 rabbitmq_peer_discovery_aws-3.7.8.ez -rw-r--r-- 1 root root 54698 Sep 20 22:35 rabbitmq_peer_discovery_common-3.7.8.ez -rw-r--r-- 1 root root 50486 Sep 20 22:35 rabbitmq_peer_discovery_consul-3.7.8.ez -rw-r--r-- 1 root root 33629 Sep 20 22:35 rabbitmq_peer_discovery_etcd-3.7.8.ez -rw-r--r-- 1 root root 14986 Sep 20 22:35 rabbitmq_peer_discovery_k8s-3.7.8.ez -rw-r--r-- 1 root root 7684 Sep 20 22:35 rabbitmq_random_exchange-3.7.8.ez -rw-r--r-- 1 root root 19130 Sep 20 22:35 rabbitmq_recent_history_exchange-3.7.8.ez -rw-r--r-- 1 root root 49768 Sep 20 22:35 rabbitmq_sharding-3.7.8.ez -rw-r--r-- 1 root root 178108 Sep 20 22:35 rabbitmq_shovel-3.7.8.ez -rw-r--r-- 1 root root 23967 Sep 20 22:35 rabbitmq_shovel_management-3.7.8.ez -rw-r--r-- 1 root root 147220 Sep 20 22:35 rabbitmq_stomp-3.7.8.ez -rw-r--r-- 1 root root 72022 Sep 20 22:35 rabbitmq_top-3.7.8.ez -rw-r--r-- 1 root root 65402 Sep 20 22:35 rabbitmq_tracing-3.7.8.ez -rw-r--r-- 1 root root 71625 Sep 20 22:35 rabbitmq_trust_store-3.7.8.ez -rw-r--r-- 1 root root 45807 Sep 20 22:35 rabbitmq_web_dispatch-3.7.8.ez -rw-r--r-- 1 root root 39338 Sep 20 22:35 rabbitmq_web_mqtt-3.7.8.ez -rw-r--r-- 1 root root 66310 Sep 20 22:35 rabbitmq_web_mqtt_examples-3.7.8.ez -rw-r--r-- 1 root root 52993 Sep 20 22:35 rabbitmq_web_stomp-3.7.8.ez -rw-r--r-- 1 root root 51996 Sep 20 22:35 rabbitmq_web_stomp_examples-3.7.8.ez -rw-r--r-- 1 root root 58340 Sep 20 22:35 ranch-1.5.0.ez -rw-r--r-- 1 root root 36698 Sep 20 22:35 ranch_proxy_protocol-1.5.0.ez -rw-r--r-- 1 root root 59 Sep 20 22:35 README -rw-r--r-- 1 root root 55911 Sep 20 22:35 recon-2.3.2.ez -rw-r--r-- 1 root root 55827 Sep 20 22:35 syslog-3.4.3.ez

noxdafox commented 6 years ago

You need to replace both plugins rabbitmq_message_deduplication-0.3.4-erl-20.ez and elixir-1.7.2-erl-20.ez.

khanhpv commented 6 years ago

You can see it appears both in the same folder (look at the date): -rw-r--r-- 1 root root 3116863 Oct 16 13:06 elixir-1.7.3-erl-20.ez -rw-r--r-- 1 root root 43816 Oct 16 13:06 rabbitmq_message_deduplication-0.3.4-erl-20.ez

It means, I have just updated it

noxdafox commented 6 years ago

Have you restarted the Broker after replacing the plugin files? I am not sure you can simply replace a plugin file while the broker is running.

Could you provide the SHA hashes of the two plugin files?

khanhpv commented 6 years ago

I have restarted and the error has gone. I'm going to test the deduplication function. Thanks for your supports.

zamnuts commented 6 years ago

Just pulled the latest ez files and put them through the same tests from OP above. Loading the plugin was successful, but crashed when declaring the queue.

sha1sum for verification:

09c8170a363479d585b73831d29481bc8df7f8ee  elixir-1.7.3-erl-20.ez
1e4e4c3d6e5680dd5f05220529c536f7367cfed1  rabbitmq_message_deduplication-0.3.4-erl-20.ez

updated Dockerfile:

FROM rabbitmq:3.7.8-management-alpine
ADD \
  rabbitmq_message_deduplication-0.3.4-erl-20.ez \
  elixir-1.7.3-erl-20.ez \
  /opt/rabbitmq/plugins/

declaring the queue (client):

bash-4.4# rabbitmqadmin -u guest -p guest declare queue name=deduptest0 'arguments={"x-message-deduplication": true}'
*** Request to node rabbit@0b17b768a154 failed with {'EXIT',
                                                 {{noproc,
                                                   {'Elixir.GenServer',call,
                                                    ['Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor',
                                                     {start_child,
                                                      {{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',
                                                        start_link,
                                                        [cache_queue__deduptest0,
                                                         [{ttl,nil},
                                                          {persistence,
                                                           memory}]]},
                                                       permanent,5000,worker,
                                                       ['Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache']}},
                                                     infinity]}},
                                                  {gen_server2,call,
                                                   [<0.657.0>,
                                                    {init,new},
                                                    infinity]}}}

exception trace (server):

2018-10-16T20:46:38.992923442Z 2018-10-16 20:46:38.992 [info] <0.643.0> Deduplication queues enabled, real BQ is rabbit_priority_queue
2018-10-16T20:46:38.993852004Z 2018-10-16 20:46:38.993 [info] <0.33.0> Application elixir started on node rabbit@0b17b768a154
2018-10-16T20:46:38.994349299Z 2018-10-16 20:46:38.994 [info] <0.33.0> Application rabbitmq_message_deduplication started on node rabbit@0b17b768a154
2018-10-16T20:46:38.996201386Z 2018-10-16 20:46:38.994 [info] <0.643.0> Plugins changed; enabled [elixir,rabbitmq_message_deduplication]
2018-10-16T20:47:33.171973655Z 2018-10-16 20:47:33.169 [warning] <0.654.0> FORMAT ERROR: "Declare queue error: ~s" [{'EXIT',{{noproc,{'Elixir.GenServer',call,['Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor',{start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,[{ttl,nil},{persistence,memory}]]},permanent,5000,worker,['Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache']}},infinity]}},{gen_server2,call,[<0.657.0>,{init,new},infinity]}}}]
2018-10-16T20:47:33.172151531Z 2018-10-16 20:47:33.170 [error] <0.657.0> ** Generic server <0.657.0> terminating
2018-10-16T20:47:33.172174604Z ** Last message in was {init,new}
2018-10-16T20:47:33.172194471Z ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"deduptest0">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.657.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460653532618,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
2018-10-16T20:47:33.172215110Z ** Reason for termination == 
2018-10-16T20:47:33.172224925Z ** {noproc,{'Elixir.GenServer',call,['Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor',{start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,[{ttl,nil},{persistence,memory}]]},permanent,5000,worker,['Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache']}},infinity]}}
2018-10-16T20:47:33.172235117Z 2018-10-16 20:47:33.170 [error] <0.659.0> Restarting crashed queue 'deduptest0' in vhost '/'.
2018-10-16T20:47:33.172244582Z 2018-10-16 20:47:33.170 [error] <0.657.0> CRASH REPORT Process <0.657.0> with 0 neighbours exited with reason: no such process or port in call to 'Elixir.GenServer':call('Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor', {start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,...]},...}}, infinity) in gen_server2:terminate/3 line 1166
2018-10-16T20:47:33.172255856Z 2018-10-16 20:47:33.171 [error] <0.656.0> Supervisor {<0.656.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"deduptest0">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.655.0>) at <0.657.0> exit with reason no such process or port in call to 'Elixir.GenServer':call('Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor', {start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,...]},...}}, infinity) in context child_terminated
2018-10-16T20:47:33.177000960Z 2018-10-16 20:47:33.175 [error] <0.661.0> Restarting crashed queue 'deduptest0' in vhost '/'.
2018-10-16T20:47:33.189985761Z 2018-10-16 20:47:33.175 [error] <0.659.0> ** Generic server <0.659.0> terminating
2018-10-16T20:47:33.190029495Z ** Last message in was {'$gen_cast',init}
2018-10-16T20:47:33.190043663Z ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"deduptest0">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.659.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460653528276,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
2018-10-16T20:47:33.190064479Z ** Reason for termination == 
2018-10-16T20:47:33.190082124Z ** {noproc,{'Elixir.GenServer',call,['Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor',{start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,[{ttl,nil},{persistence,memory}]]},permanent,5000,worker,['Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache']}},infinity]}}
2018-10-16T20:47:33.190094268Z 2018-10-16 20:47:33.176 [error] <0.659.0> CRASH REPORT Process <0.659.0> with 0 neighbours exited with reason: no such process or port in call to 'Elixir.GenServer':call('Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor', {start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,...]},...}}, infinity) in gen_server2:terminate/3 line 1166
2018-10-16T20:47:33.190106853Z 2018-10-16 20:47:33.176 [error] <0.656.0> Supervisor {<0.656.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"deduptest0">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.655.0>) at <0.659.0> exit with reason no such process or port in call to 'Elixir.GenServer':call('Elixir.RabbitMQ.MessageDeduplicationPlugin.Supervisor', {start_child,{{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Cache',start_link,[cache_queue__deduptest0,...]},...}}, infinity) in context child_terminated
2018-10-16T20:47:33.190119063Z 2018-10-16 20:47:33.183 [error] <0.663.0> Restarting crashed queue 'deduptest0' in vhost '/'.
2018-10-16T20:47:33.190127530Z 2018-10-16 20:47:33.183 [error] <0.661.0> ** Generic server <0.661.0> terminating

... this just keeps repeating

noxdafox commented 6 years ago

Did you restart the broker after installing the plugin?

To install the plugin you need to:

zamnuts commented 6 years ago

@noxdafox, this was a fresh container instance with a new image build, so yes, it was restarted. The plug-in loaded this time as can be seen from the logs.

noxdafox commented 6 years ago

This is curious as the crash log looks exactly as the one in issue #16. The crash indicates that the cache supervisor process is not running at the time you try to create the queue. This means the application plugin was not started properly when the broker came up.

I will try to reproduce this in a Docker container.

noxdafox commented 6 years ago

Just reproduced your issue and my assumptions are right.

The rabbitmq-message-deduplication plugins requires the broker to be restarted once installed and enabled. This is due to the fact the plugin relies on RabbitMQ boot sequence to start its own supervision process.

Just issue a docker restart <container-id> and the problem will be solved.

noxdafox commented 6 years ago

I discovered why it was necessary to restart RabbitMQ to take the plugin into use. It was a faulty boot step registration.

This is handled in commit a9417a2 and will be soon released as a fix in version 0.3.5.

zamnuts commented 6 years ago

When starting rabbitmq with this deduplication plugin defined in /etc/rabbitmq/enabled_plugins (extrapolating from your description, this should not require a broker restart), deduplication works! 🎉

enabled_plugins:

[rabbitmq_management,rabbitmq_management_agent,
  rabbitmq_shovel,rabbitmq_shovel_management,
  rabbitmq_message_deduplication].

server log:

2018-10-17T21:28:49.076697486Z 2018-10-17 21:28:49.076 [debug] <0.739.0> Supervisor {<0.739.0>,rabbit_amqqueue_sup} started rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"deduptest0">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.738.0>) at pid <0.740.0>
2018-10-17T21:28:49.079616896Z 2018-10-17 21:28:49.079 [debug] <0.740.0> Starting queue deduplication cache cache_queue__deduptest0 with options [{ttl,nil},{persistence,memory}]

Using this configuration would be more typical in a container service setup (k8s/swarm), vs enabling plugins via runtime (config files vs scripted commands).

Despite this workaround being successful, I will upgrade when v0.3.5 is out to cover potential config edge cases. Many thanks for your help and prompt responses. 👍

noxdafox commented 6 years ago

Version 0.3.5 released.

urkl commented 6 years ago

This is my solution for enabling plugins for offical rabbitmq container on docker hub:

docker-compose.yml: 

  rabbitmq:
    image: rabbitmq:management
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      - RABBITMQ_DEFAULT_USER=rabbit
      - RABBITMQ_DEFAULT_PASS=rabbit
      - TZ=Europe/Ljubljana
    volumes:
    - ./rabbit/enabled_plugins:/etc/rabbitmq/enabled_plugins
    - ./rabbit/plugins:/usr/lib/rabbitmq/plugins

Enabled plugins file contents:

[rabbitmq_management,rabbitmq_message_deduplication].

And local ./rabbit/plugins folders contains elixir and deduplication plugin.