search

rdnfn / beobench

A toolkit providing easy and unified access to building control environments for reinforcement learning (RL).
https://beobench.readthedocs.io
MIT License
37 stars 4 forks source link

Add minimal example for splitting environment and agent into two containers. #92

Closed david-woelfle closed 1 year ago

david-woelfle commented 1 year ago

Works with identical python versions for now.

This could be a first approach to improve issue #86

rdnfn commented 1 year ago

Thanks a lot for adding this example @david-woelfle! Looking into getting this up and running, and then extending it now!

rdnfn commented 1 year ago

Thanks a lot for adding the PR, I merged it with a new feat/split-env-agent branch.

Just tried to run your code in the Beobench devcontainer and I am getting the connection refused error below in the agent worker. Any idea why this might be these case?

beobench-rabitmq-broker     | 2022-10-24 09:55:49.975500+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
beobench-agent-dummy        | Traceback (most recent call last):
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/utils/functional.py", line 30, in __call__
beobench-agent-dummy        |     return self.__value__
beobench-agent-dummy        | AttributeError: 'ChannelPromise' object has no attribute '__value__'. Did you mean: '__call__'?
beobench-agent-dummy        | 
beobench-agent-dummy        | During handling of the above exception, another exception occurred:
beobench-agent-dummy        | 
beobench-agent-dummy        | Traceback (most recent call last):
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
beobench-agent-dummy        |     yield
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 433, in _ensure_connection
beobench-agent-dummy        |     return retry_over_time(
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
beobench-agent-dummy        |     return fun(*args, **kwargs)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 877, in _connection_factory
beobench-agent-dummy        |     self._connection = self._establish_connection()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 812, in _establish_connection
beobench-agent-dummy        |     conn = self.transport.establish_connection()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
beobench-agent-dummy        |     conn.connect()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/amqp/connection.py", line 323, in connect
beobench-agent-dummy        |     self.transport.connect()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/amqp/transport.py", line 129, in connect
beobench-agent-dummy        |     self._connect(self.host, self.port, self.connect_timeout)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/amqp/transport.py", line 184, in _connect
beobench-agent-dummy        |     self.sock.connect(sa)
beobench-agent-dummy        | ConnectionRefusedError: [Errno 111] Connection refused
beobench-agent-dummy        | 
beobench-agent-dummy        | The above exception was the direct cause of the following exception:
beobench-agent-dummy        | 
beobench-agent-dummy        | Traceback (most recent call last):
beobench-agent-dummy        |   File "/source/agent.py", line 12, in <module>
beobench-agent-dummy        |     async_result = reset.delay()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 425, in delay
beobench-agent-dummy        |     return self.apply_async(args, kwargs)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 575, in apply_async
beobench-agent-dummy        |     return app.send_task(
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/app/base.py", line 787, in send_task
beobench-agent-dummy        |     self.backend.on_task_call(P, task_id)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/backends/rpc.py", line 164, in on_task_call
beobench-agent-dummy        |     maybe_declare(self.binding(producer.channel), retry=True)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 209, in _get_channel
beobench-agent-dummy        |     channel = self._channel = channel()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/utils/functional.py", line 32, in __call__
beobench-agent-dummy        |     value = self.__value__ = self.__contract__()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 225, in <lambda>
beobench-agent-dummy        |     channel = ChannelPromise(lambda: connection.default_channel)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 895, in default_channel
beobench-agent-dummy        |     self._ensure_connection(**conn_opts)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 432, in _ensure_connection
beobench-agent-dummy        |     with ctx():
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
beobench-agent-dummy        |     self.gen.throw(typ, value, traceback)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
beobench-agent-dummy        |     raise ConnectionError(str(exc)) from exc
beobench-agent-dummy        | kombu.exceptions.OperationalError: [Errno 111] Connection refused

Full trace:

root ➜ /beobench/split_containers_example (feat/split-env-agent ✗) $ docker compose up
[+] Running 3/0
 ⠿ Container beobench-rabitmq-broker     Recreated                                                                                                                                                           0.0s
 ⠿ Container beobench-agent-dummy        Created                                                                                                                                                             0.0s
 ⠿ Container beobench-environment-dummy  Created                                                                                                                                                             0.0s
Attaching to beobench-agent-dummy, beobench-environment-dummy, beobench-rabitmq-broker
beobench-environment-dummy  | /usr/local/lib/python3.10/site-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
beobench-environment-dummy  | absolutely not recommended!
beobench-environment-dummy  | 
beobench-environment-dummy  | Please specify a different user using the --uid option.
beobench-environment-dummy  | 
beobench-environment-dummy  | User information: uid=0 euid=0 gid=0 egid=0
beobench-environment-dummy  | 
beobench-environment-dummy  |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
beobench-environment-dummy  |  
beobench-environment-dummy  |  -------------- celery@da698d306446 v5.2.7 (dawn-chorus)
beobench-environment-dummy  | --- ***** ----- 
beobench-environment-dummy  | -- ******* ---- Linux-5.10.76-linuxkit-aarch64-with-glibc2.31 2022-10-24 09:55:44
beobench-environment-dummy  | - *** --- * --- 
beobench-environment-dummy  | - ** ---------- [config]
beobench-environment-dummy  | - ** ---------- .> app:         remote-environment:0xffff94a6d1e0
beobench-environment-dummy  | - ** ---------- .> transport:   amqp://guest:**@beobench-rabitmq-broker:5672//
beobench-environment-dummy  | - ** ---------- .> results:     rpc://
beobench-environment-dummy  | - *** --- * --- .> concurrency: 4 (prefork)
beobench-environment-dummy  | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
beobench-environment-dummy  | --- ***** ----- 
beobench-environment-dummy  |  -------------- [queues]
beobench-environment-dummy  |                 .> celery           exchange=celery(direct) key=celery
beobench-environment-dummy  |                 
beobench-environment-dummy  | 
beobench-environment-dummy  | [tasks]
beobench-environment-dummy  |   . envwrapper.reset
beobench-environment-dummy  |   . envwrapper.step
beobench-environment-dummy  | 
beobench-environment-dummy  | [2022-10-24 09:55:44,123: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@beobench-rabitmq-broker:5672//: [Errno 111] Connection refused.
beobench-environment-dummy  | Trying again in 2.00 seconds... (1/100)
beobench-environment-dummy  | 
beobench-environment-dummy  | [2022-10-24 09:55:46,127: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@beobench-rabitmq-broker:5672//: [Errno 111] Connection refused.
beobench-environment-dummy  | Trying again in 4.00 seconds... (2/100)
beobench-environment-dummy  | 
beobench-rabitmq-broker     | 2022-10-24 09:55:46.971852+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
beobench-rabitmq-broker     | 2022-10-24 09:55:46.975991+00:00 [notice] <0.229.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
beobench-rabitmq-broker     | 2022-10-24 09:55:46.992787+00:00 [notice] <0.229.0> Logging: configured log handlers are now ACTIVE
beobench-rabitmq-broker     | 2022-10-24 09:55:47.195745+00:00 [info] <0.229.0> ra: starting system quorum_queues
beobench-rabitmq-broker     | 2022-10-24 09:55:47.195801+00:00 [info] <0.229.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@11ec45d2f3c1/quorum/rabbit@11ec45d2f3c1
beobench-rabitmq-broker     | 2022-10-24 09:55:47.258271+00:00 [info] <0.265.0> ra system 'quorum_queues' running pre init for 0 registered servers
beobench-rabitmq-broker     | 2022-10-24 09:55:47.269411+00:00 [info] <0.266.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
beobench-rabitmq-broker     | 2022-10-24 09:55:47.292695+00:00 [notice] <0.271.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
beobench-rabitmq-broker     | 2022-10-24 09:55:47.301642+00:00 [info] <0.229.0> ra: starting system coordination
beobench-rabitmq-broker     | 2022-10-24 09:55:47.301692+00:00 [info] <0.229.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@11ec45d2f3c1/coordination/rabbit@11ec45d2f3c1
beobench-rabitmq-broker     | 2022-10-24 09:55:47.303050+00:00 [info] <0.278.0> ra system 'coordination' running pre init for 0 registered servers
beobench-rabitmq-broker     | 2022-10-24 09:55:47.304322+00:00 [info] <0.279.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
beobench-rabitmq-broker     | 2022-10-24 09:55:47.304447+00:00 [notice] <0.284.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
beobench-rabitmq-broker     | 2022-10-24 09:55:47.309142+00:00 [info] <0.229.0> 
beobench-rabitmq-broker     | 2022-10-24 09:55:47.309142+00:00 [info] <0.229.0>  Starting RabbitMQ 3.11.2 on Erlang 25.1.1 [jit]
beobench-rabitmq-broker     | 2022-10-24 09:55:47.309142+00:00 [info] <0.229.0>  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
beobench-rabitmq-broker     | 2022-10-24 09:55:47.309142+00:00 [info] <0.229.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
beobench-rabitmq-broker     | 
beobench-rabitmq-broker     |   ##  ##      RabbitMQ 3.11.2
beobench-rabitmq-broker     |   ##  ##
beobench-rabitmq-broker     |   ##########  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
beobench-rabitmq-broker     |   ######  ##
beobench-rabitmq-broker     |   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
beobench-rabitmq-broker     | 
beobench-rabitmq-broker     |   Erlang:      25.1.1 [jit]
beobench-rabitmq-broker     |   TLS Library: OpenSSL - OpenSSL 1.1.1q  5 Jul 2022
beobench-rabitmq-broker     |   Release series support status: supported
beobench-rabitmq-broker     | 
beobench-rabitmq-broker     |   Doc guides:  https://rabbitmq.com/documentation.html
beobench-rabitmq-broker     |   Support:     https://rabbitmq.com/contact.html
beobench-rabitmq-broker     |   Tutorials:   https://rabbitmq.com/getstarted.html
beobench-rabitmq-broker     |   Monitoring:  https://rabbitmq.com/monitoring.html
beobench-rabitmq-broker     | 
beobench-rabitmq-broker     |   Logs: /var/log/rabbitmq/rabbit@11ec45d2f3c1_upgrade.log
beobench-rabitmq-broker     |         <stdout>
beobench-rabitmq-broker     | 
beobench-rabitmq-broker     |   Config file(s): /etc/rabbitmq/conf.d/10-defaults.conf
beobench-rabitmq-broker     |                   /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
beobench-rabitmq-broker     | 
beobench-rabitmq-broker     |   Starting broker...2022-10-24 09:55:47.313712+00:00 [info] <0.229.0> 
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>  node           : rabbit@11ec45d2f3c1
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>  home dir       : /var/lib/rabbitmq
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>  config file(s) : /etc/rabbitmq/conf.d/10-defaults.conf
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>                 : /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>  cookie hash    : usBLDO54SOhZUDgn/70vfQ==
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>  log(s)         : /var/log/rabbitmq/rabbit@11ec45d2f3c1_upgrade.log
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>                 : <stdout>
beobench-rabitmq-broker     | 2022-10-24 09:55:47.313712+00:00 [info] <0.229.0>  database dir   : /var/lib/rabbitmq/mnesia/rabbit@11ec45d2f3c1
beobench-rabitmq-broker     | 2022-10-24 09:55:49.950333+00:00 [info] <0.229.0> Running boot step pre_boot defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.950426+00:00 [info] <0.229.0> Running boot step rabbit_global_counters defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.950704+00:00 [info] <0.229.0> Running boot step rabbit_osiris_metrics defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.950839+00:00 [info] <0.229.0> Running boot step rabbit_core_metrics defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.951490+00:00 [info] <0.229.0> Running boot step rabbit_alarm defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.955723+00:00 [info] <0.298.0> Memory high watermark set to 1573 MiB (1649747558 bytes) of 3933 MiB (4124368896 bytes) total
beobench-rabitmq-broker     | 2022-10-24 09:55:49.959030+00:00 [info] <0.300.0> Enabling free disk space monitoring
beobench-rabitmq-broker     | 2022-10-24 09:55:49.959062+00:00 [info] <0.300.0> Disk free limit set to 50MB
beobench-rabitmq-broker     | 2022-10-24 09:55:49.960820+00:00 [info] <0.229.0> Running boot step code_server_cache defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.961071+00:00 [info] <0.229.0> Running boot step file_handle_cache defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.961281+00:00 [info] <0.303.0> Limiting to approx 1048479 file handles (943629 sockets)
beobench-rabitmq-broker     | 2022-10-24 09:55:49.961397+00:00 [info] <0.304.0> FHC read buffering: OFF
beobench-rabitmq-broker     | 2022-10-24 09:55:49.961446+00:00 [info] <0.304.0> FHC write buffering: ON
beobench-rabitmq-broker     | 2022-10-24 09:55:49.964504+00:00 [info] <0.229.0> Running boot step worker_pool defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.965302+00:00 [info] <0.286.0> Will use 4 processes for default worker pool
beobench-rabitmq-broker     | 2022-10-24 09:55:49.965338+00:00 [info] <0.286.0> Starting worker pool 'worker_pool' with 4 processes in it
beobench-rabitmq-broker     | 2022-10-24 09:55:49.965626+00:00 [info] <0.229.0> Running boot step database defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:49.967243+00:00 [info] <0.229.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@11ec45d2f3c1 is empty. Assuming we need to join an existing cluster or initialise from scratch...
beobench-rabitmq-broker     | 2022-10-24 09:55:49.967292+00:00 [info] <0.229.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
beobench-rabitmq-broker     | 2022-10-24 09:55:49.967320+00:00 [info] <0.229.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
beobench-rabitmq-broker     | 2022-10-24 09:55:49.967404+00:00 [info] <0.229.0> All discovered existing cluster peers:
beobench-rabitmq-broker     | 2022-10-24 09:55:49.967427+00:00 [info] <0.229.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
beobench-rabitmq-broker     | 2022-10-24 09:55:49.975500+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
beobench-agent-dummy        | Traceback (most recent call last):
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/utils/functional.py", line 30, in __call__
beobench-agent-dummy        |     return self.__value__
beobench-agent-dummy        | AttributeError: 'ChannelPromise' object has no attribute '__value__'. Did you mean: '__call__'?
beobench-agent-dummy        | 
beobench-agent-dummy        | During handling of the above exception, another exception occurred:
beobench-agent-dummy        | 
beobench-agent-dummy        | Traceback (most recent call last):
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
beobench-agent-dummy        |     yield
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 433, in _ensure_connection
beobench-agent-dummy        |     return retry_over_time(
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
beobench-agent-dummy        |     return fun(*args, **kwargs)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 877, in _connection_factory
beobench-agent-dummy        |     self._connection = self._establish_connection()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 812, in _establish_connection
beobench-agent-dummy        |     conn = self.transport.establish_connection()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
beobench-agent-dummy        |     conn.connect()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/amqp/connection.py", line 323, in connect
beobench-agent-dummy        |     self.transport.connect()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/amqp/transport.py", line 129, in connect
beobench-agent-dummy        |     self._connect(self.host, self.port, self.connect_timeout)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/amqp/transport.py", line 184, in _connect
beobench-agent-dummy        |     self.sock.connect(sa)
beobench-agent-dummy        | ConnectionRefusedError: [Errno 111] Connection refused
beobench-agent-dummy        | 
beobench-agent-dummy        | The above exception was the direct cause of the following exception:
beobench-agent-dummy        | 
beobench-agent-dummy        | Traceback (most recent call last):
beobench-agent-dummy        |   File "/source/agent.py", line 12, in <module>
beobench-agent-dummy        |     async_result = reset.delay()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 425, in delay
beobench-agent-dummy        |     return self.apply_async(args, kwargs)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 575, in apply_async
beobench-agent-dummy        |     return app.send_task(
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/app/base.py", line 787, in send_task
beobench-agent-dummy        |     self.backend.on_task_call(P, task_id)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/celery/backends/rpc.py", line 164, in on_task_call
beobench-agent-dummy        |     maybe_declare(self.binding(producer.channel), retry=True)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 209, in _get_channel
beobench-agent-dummy        |     channel = self._channel = channel()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/utils/functional.py", line 32, in __call__
beobench-agent-dummy        |     value = self.__value__ = self.__contract__()
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/messaging.py", line 225, in <lambda>
beobench-agent-dummy        |     channel = ChannelPromise(lambda: connection.default_channel)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 895, in default_channel
beobench-agent-dummy        |     self._ensure_connection(**conn_opts)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 432, in _ensure_connection
beobench-agent-dummy        |     with ctx():
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
beobench-agent-dummy        |     self.gen.throw(typ, value, traceback)
beobench-agent-dummy        |   File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
beobench-agent-dummy        |     raise ConnectionError(str(exc)) from exc
beobench-agent-dummy        | kombu.exceptions.OperationalError: [Errno 111] Connection refused
beobench-rabitmq-broker     | 2022-10-24 09:55:50.111568+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
beobench-rabitmq-broker     | 2022-10-24 09:55:50.111757+00:00 [info] <0.229.0> Successfully synced tables from a peer
beobench-rabitmq-broker     | 2022-10-24 09:55:50.120900+00:00 [info] <0.229.0> Feature flags: `feature_flags_v2`: supported, attempt to enable...
beobench-environment-dummy  | [2022-10-24 09:55:50,135: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@beobench-rabitmq-broker:5672//: [Errno 111] Connection refused.
beobench-environment-dummy  | Trying again in 6.00 seconds... (3/100)
beobench-environment-dummy  | 
beobench-rabitmq-broker     | 2022-10-24 09:55:50.145365+00:00 [notice] <0.287.0> Feature flags: attempt to enable `classic_mirrored_queue_version`...
beobench-agent-dummy exited with code 1
beobench-rabitmq-broker     | 2022-10-24 09:55:50.176769+00:00 [notice] <0.287.0> Feature flags: `classic_mirrored_queue_version` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.177241+00:00 [notice] <0.287.0> Feature flags: attempt to enable `classic_queue_type_delivery_support`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.185733+00:00 [notice] <0.287.0> Feature flags: attempt to enable `stream_queue`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.204742+00:00 [notice] <0.287.0> Feature flags: `stream_queue` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.216858+00:00 [notice] <0.287.0> Feature flags: `classic_queue_type_delivery_support` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.217420+00:00 [notice] <0.287.0> Feature flags: attempt to enable `direct_exchange_routing_v2`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.225439+00:00 [info] <0.497.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
beobench-rabitmq-broker     | 2022-10-24 09:55:50.225631+00:00 [info] <0.497.0> Successfully synced tables from a peer
beobench-rabitmq-broker     | 2022-10-24 09:55:50.242266+00:00 [notice] <0.287.0> Feature flags: `direct_exchange_routing_v2` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.242850+00:00 [notice] <0.287.0> Feature flags: attempt to enable `drop_unroutable_metric`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.261992+00:00 [notice] <0.287.0> Feature flags: `drop_unroutable_metric` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.262405+00:00 [notice] <0.287.0> Feature flags: attempt to enable `empty_basic_get_metric`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.282243+00:00 [notice] <0.287.0> Feature flags: `empty_basic_get_metric` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.282834+00:00 [notice] <0.287.0> Feature flags: attempt to enable `listener_records_in_ets`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.315553+00:00 [notice] <0.287.0> Feature flags: `listener_records_in_ets` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.316357+00:00 [notice] <0.287.0> Feature flags: attempt to enable `stream_single_active_consumer`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.334310+00:00 [notice] <0.287.0> Feature flags: `stream_single_active_consumer` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.334857+00:00 [notice] <0.287.0> Feature flags: attempt to enable `tracking_records_in_ets`...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.353144+00:00 [notice] <0.287.0> Feature flags: `tracking_records_in_ets` enabled
beobench-rabitmq-broker     | 2022-10-24 09:55:50.353605+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
beobench-rabitmq-broker     | 2022-10-24 09:55:50.353706+00:00 [info] <0.229.0> Successfully synced tables from a peer
beobench-rabitmq-broker     | 2022-10-24 09:55:50.363891+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364085+00:00 [info] <0.229.0> Successfully synced tables from a peer
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364125+00:00 [info] <0.229.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364154+00:00 [info] <0.229.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364259+00:00 [info] <0.229.0> Running boot step tracking_metadata_store defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364406+00:00 [info] <0.619.0> Setting up a table for connection tracking on this node: tracked_connection
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364452+00:00 [info] <0.619.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364486+00:00 [info] <0.619.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364536+00:00 [info] <0.619.0> Setting up a table for channel tracking on this node: tracked_channel
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364613+00:00 [info] <0.619.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364703+00:00 [info] <0.229.0> Running boot step networking_metadata_store defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364765+00:00 [info] <0.229.0> Running boot step database_sync defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.364808+00:00 [info] <0.229.0> Running boot step feature_flags defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365282+00:00 [info] <0.229.0> Running boot step codec_correctness_check defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365324+00:00 [info] <0.229.0> Running boot step external_infrastructure defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365349+00:00 [info] <0.229.0> Running boot step rabbit_event defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365426+00:00 [info] <0.229.0> Running boot step rabbit_registry defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365485+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365550+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365587+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365612+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365643+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365692+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365717+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365736+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365844+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.365994+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366037+00:00 [info] <0.229.0> Running boot step rabbit_priority_queue defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366056+00:00 [info] <0.229.0> Priority queues enabled, real BQ is rabbit_variable_queue
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366111+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366197+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366238+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_random defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366293+00:00 [info] <0.229.0> Running boot step kernel_ready defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366313+00:00 [info] <0.229.0> Running boot step rabbit_sysmon_minder defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.366395+00:00 [info] <0.229.0> Running boot step rabbit_epmd_monitor defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.367344+00:00 [info] <0.628.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
beobench-rabitmq-broker     | 2022-10-24 09:55:50.367405+00:00 [info] <0.229.0> Running boot step guid_generator defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.369092+00:00 [info] <0.229.0> Running boot step rabbit_node_monitor defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.369646+00:00 [info] <0.632.0> Starting rabbit_node_monitor
beobench-rabitmq-broker     | 2022-10-24 09:55:50.369848+00:00 [info] <0.229.0> Running boot step delegate_sup defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.370281+00:00 [info] <0.229.0> Running boot step rabbit_memory_monitor defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.370380+00:00 [info] <0.229.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.370448+00:00 [info] <0.229.0> Running boot step core_initialized defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.370484+00:00 [info] <0.229.0> Running boot step upgrade_queues defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.376242+00:00 [info] <0.229.0> message_store upgrades: 1 to apply
beobench-rabitmq-broker     | 2022-10-24 09:55:50.376357+00:00 [info] <0.229.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
beobench-rabitmq-broker     | 2022-10-24 09:55:50.376462+00:00 [info] <0.229.0> message_store upgrades: No durable queues found. Skipping message store migration
beobench-rabitmq-broker     | 2022-10-24 09:55:50.376497+00:00 [info] <0.229.0> message_store upgrades: Removing the old message store data
beobench-rabitmq-broker     | 2022-10-24 09:55:50.379379+00:00 [info] <0.229.0> message_store upgrades: All upgrades applied successfully
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387467+00:00 [info] <0.229.0> Running boot step channel_tracking defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387529+00:00 [info] <0.229.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387592+00:00 [info] <0.229.0> Running boot step connection_tracking defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387614+00:00 [info] <0.229.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387652+00:00 [info] <0.229.0> Running boot step rabbit_definitions_hashing defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387723+00:00 [info] <0.229.0> Running boot step rabbit_exchange_parameters defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.387784+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388060+00:00 [info] <0.229.0> Running boot step rabbit_policies defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388252+00:00 [info] <0.229.0> Running boot step rabbit_policy defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388303+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_validator defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388327+00:00 [info] <0.229.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388352+00:00 [info] <0.229.0> Running boot step rabbit_stream_coordinator defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388444+00:00 [info] <0.229.0> Running boot step rabbit_vhost_limit defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388496+00:00 [info] <0.229.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388516+00:00 [info] <0.229.0> Management plugin: using rates mode 'basic'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.388971+00:00 [info] <0.229.0> Running boot step recovery defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.389723+00:00 [info] <0.229.0> Running boot step empty_db_check defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.389769+00:00 [info] <0.229.0> Will seed default virtual host and user...
beobench-rabitmq-broker     | 2022-10-24 09:55:50.389848+00:00 [info] <0.229.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
beobench-rabitmq-broker     | 2022-10-24 09:55:50.403119+00:00 [info] <0.674.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@11ec45d2f3c1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
beobench-rabitmq-broker     | 2022-10-24 09:55:50.404485+00:00 [info] <0.674.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.408333+00:00 [info] <0.674.0> Starting message stores for vhost '/'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.408667+00:00 [info] <0.679.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
beobench-rabitmq-broker     | 2022-10-24 09:55:50.410234+00:00 [info] <0.674.0> Started message store of type transient for vhost '/'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.410383+00:00 [info] <0.683.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
beobench-rabitmq-broker     | 2022-10-24 09:55:50.412068+00:00 [warning] <0.683.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
beobench-rabitmq-broker     | 2022-10-24 09:55:50.414512+00:00 [info] <0.674.0> Started message store of type persistent for vhost '/'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.414688+00:00 [info] <0.674.0> Recovering 0 queues of type rabbit_classic_queue took 9ms
beobench-rabitmq-broker     | 2022-10-24 09:55:50.414727+00:00 [info] <0.674.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
beobench-rabitmq-broker     | 2022-10-24 09:55:50.414766+00:00 [info] <0.674.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
beobench-rabitmq-broker     | 2022-10-24 09:55:50.417840+00:00 [info] <0.229.0> Created user 'guest'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.419106+00:00 [info] <0.229.0> Successfully set user tags for user 'guest' to [administrator]
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420454+00:00 [info] <0.229.0> Successfully set permissions for 'guest' in virtual host '/' to '.*', '.*', '.*'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420526+00:00 [info] <0.229.0> Running boot step rabbit_observer_cli defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420564+00:00 [info] <0.229.0> Running boot step rabbit_looking_glass defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420594+00:00 [info] <0.229.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420748+00:00 [info] <0.229.0> Running boot step background_gc defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420823+00:00 [info] <0.229.0> Running boot step routing_ready defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420839+00:00 [info] <0.229.0> Running boot step pre_flight defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420851+00:00 [info] <0.229.0> Running boot step notify_cluster defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420870+00:00 [info] <0.229.0> Running boot step networking defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420884+00:00 [info] <0.229.0> Running boot step definition_import_worker_pool defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.420982+00:00 [info] <0.286.0> Starting worker pool 'definition_import_pool' with 4 processes in it
beobench-rabitmq-broker     | 2022-10-24 09:55:50.421206+00:00 [info] <0.229.0> Running boot step cluster_name defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.421269+00:00 [info] <0.229.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-WNGDlaIalpl4tgB0MDNKLg'
beobench-rabitmq-broker     | 2022-10-24 09:55:50.422619+00:00 [info] <0.229.0> Running boot step direct_client defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.422690+00:00 [info] <0.229.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
beobench-rabitmq-broker     | 2022-10-24 09:55:50.422708+00:00 [info] <0.229.0> Creating table rabbit_node_maintenance_states for maintenance mode status
beobench-rabitmq-broker     | 2022-10-24 09:55:50.428079+00:00 [info] <0.720.0> Resetting node maintenance status
beobench-rabitmq-broker     | 2022-10-24 09:55:50.438604+00:00 [info] <0.747.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
beobench-rabitmq-broker     | 2022-10-24 09:55:50.438743+00:00 [info] <0.720.0> Ready to start client connection listeners
beobench-rabitmq-broker     | 2022-10-24 09:55:50.440089+00:00 [info] <0.791.0> started TCP listener on [::]:5672
beobench-rabitmq-broker     |  completed with 3 plugins.
beobench-rabitmq-broker     | 2022-10-24 09:55:50.563484+00:00 [info] <0.720.0> Server startup complete; 3 plugins started.
beobench-rabitmq-broker     | 2022-10-24 09:55:50.563484+00:00 [info] <0.720.0>  * rabbitmq_prometheus
beobench-rabitmq-broker     | 2022-10-24 09:55:50.563484+00:00 [info] <0.720.0>  * rabbitmq_web_dispatch
beobench-rabitmq-broker     | 2022-10-24 09:55:50.563484+00:00 [info] <0.720.0>  * rabbitmq_management_agent
beobench-rabitmq-broker     | 2022-10-24 09:55:56.158859+00:00 [info] <0.794.0> accepting AMQP connection <0.794.0> (172.19.0.2:49184 -> 172.19.0.3:5672)
beobench-rabitmq-broker     | 2022-10-24 09:55:56.168081+00:00 [info] <0.794.0> connection <0.794.0> (172.19.0.2:49184 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
beobench-environment-dummy  | [2022-10-24 09:55:56,169: INFO/MainProcess] Connected to amqp://guest:**@beobench-rabitmq-broker:5672//
beobench-rabitmq-broker     | 2022-10-24 09:55:56.175670+00:00 [info] <0.802.0> accepting AMQP connection <0.802.0> (172.19.0.2:49186 -> 172.19.0.3:5672)
beobench-rabitmq-broker     | 2022-10-24 09:55:56.182548+00:00 [info] <0.802.0> connection <0.802.0> (172.19.0.2:49186 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
beobench-environment-dummy  | [2022-10-24 09:55:56,183: INFO/MainProcess] mingle: searching for neighbors
beobench-rabitmq-broker     | 2022-10-24 09:55:56.203530+00:00 [info] <0.820.0> accepting AMQP connection <0.820.0> (172.19.0.2:49188 -> 172.19.0.3:5672)
beobench-rabitmq-broker     | 2022-10-24 09:55:56.206592+00:00 [info] <0.820.0> connection <0.820.0> (172.19.0.2:49188 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
beobench-environment-dummy  | [2022-10-24 09:55:57,222: INFO/MainProcess] mingle: all alone
beobench-environment-dummy  | [2022-10-24 09:55:57,261: INFO/MainProcess] celery@da698d306446 ready.
david-woelfle commented 1 year ago

Seems like the agent tried to interact with the broker before the broker was online. The connection between the environment and the broker worked, although only on the second attempt. We can discuss possible solutions later offline.

rdnfn commented 1 year ago

Sounds good! Managed to fix it by adding depends_on: ["beobench-rabbitmq-broker"] to both the env and agent container (docs) in the compose config, as in:

version: '3'

services:

  beobench-rabitmq-broker:
    container_name: beobench-rabitmq-broker
    image: rabbitmq:3
    init: true # Faster shutdown.

  beobench-environment-dummy:
    container_name: beobench-environment-dummy
    build:
      context: ./source
      dockerfile: Dockerfile_environment
    init: true # Faster shutdown.
    depends_on: [ "beobench-rabitmq-broker" ]

  beobench-agent-dummy:
    container_name: beobench-agent-dummy
    build:
      context: ./source
      dockerfile: Dockerfile_agent
    init: true # Faster shutdown.
    depends_on: [ "beobench-rabitmq-broker" ]

But let's discuss this further offline and I will add any results from this discussion here.

EDIT: note that my fix does work most of the time but not all of the time.