search

astarte-platform / astarte_vmq_plugin

Astarte VerneMQ plugin publishes incoming messages on an AMQP exchange with some additional metadata, so they can be processed by other components.
http://astarte-platform.org/
Apache License 2.0
16 stars 10 forks source link

Queue astarte_vmq_plugin_rpc disappearing from RabbitMQ #78

Open Pavinati opened 1 year ago

Pavinati commented 1 year ago

We found an odd crash on Data Updater Plant. It crashed with a timeout after trying to push a force_disconnection message in the VerneMQ RPC queue. 

Aug  7 12:48:27 |ERRO| GenServer {Registry.DataUpdater, {"test", <<110, 174, 96, 120, 180, 191, 67, 214, 138, 198, 238, 112, 135, 193, 218, 234>>}} terminating
** (stop) exited in: GenServer.call(Astarte.RPC.AMQP.Client, {:rpc, <<26, 33, 10, 29, 108, 117, 116, 101, 115, 116, 47, 98, 113, 53, 103, 101, 76, 83, 95, 81, 57, 97, 75, 120, 117, 53, 119, 104, 56, 72, 97, 54, 103, 16, 1>>, "vmq_plugin_rpc"}, 5000)
    ** (EXIT) time out
    (elixir 1.14.5) lib/gen_server.ex:1038: GenServer.call/3
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/rpc/vmq_plugin.ex:63: Astarte.DataUpdaterPlant.RPC.VMQPlugin.disconnect/2
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/impl.ex:2152: Astarte.DataUpdaterPlant.DataUpdater.Impl.force_disconnection/2
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/impl.ex:2131: Astarte.DataUpdaterPlant.DataUpdater.Impl.ask_clean_session/2
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/impl.ex:875: Astarte.DataUpdaterPlant.DataUpdater.Impl.handle_data/6
    (astarte_data_updater_plant 1.1.0) lib/astarte_data_updater_plant/data_updater/server.ex:83: Astarte.DataUpdaterPlant.DataUpdater.Server.handle_cast/2
    (stdlib 4.3.1) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.3.1) gen_server.erl:1200: :gen_server.handle_msg/6

After some debugging we found that the astarte_vmq_plugin_rpc queue was not present in RabbitMQ and even killing Astarte.RPC.AMQP.Server (which is responsible for declaring said queue) did not restore it. in the end we opted for restarting VerneMQ which solved the issue and the queue was declared as expected.

Environment: