Workers Pushing Where No Queue Exists in RabbitMQ

[skrumzy]

Environment: Docker Swarm Host: Ubuntu Server 20.04 LTS Hardware: 8 core 32GB

I'm experiencing an issue where my workers are trying to push to RabbitMQ where the referenced queue item no longer exists. This is causing massive amount of connections to RabbitMQ that essentially get dropped and recreated over and over. I'm not quite sure how to address this. I've tried:

• Deleting the referenced UUID connector container
• Deleting the connector in OpenCTI connector's page
• Resetting the connector
• Doing a Service Update to redeploy worker, RabbitMQ, and history containers

What I'm not understanding is where or what is queueing the workers to call out to RabbitMQ and how do I clear that?

WARNING:pika.channel:Received remote Channel.Close (404): "NOT_FOUND - no queue 'push_e866a073-a2ce-44d6-867e-5203738aa30d' in vhost '/'" on <Channel number=1 OPEN conn=<SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fdf532fa790> params=>> INFO:root:Thread for queue push_e866a073-a2ce-44d6-867e-5203738aa30d terminated** Exception in thread Thread-15559: Traceback (most recent call last): File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/opt/opencti-worker/worker.py", line 245, in run self.channel.basic_consume( File "/usr/local/lib/python3.9/site-packages/pika/adapters/blocking_connection.py", line 1627, in basic_consume return self._basic_consume_impl( File "/usr/local/lib/python3.9/site-packages/pika/adapters/blocking_connection.py", line 1696, in _basic_consume_impl self._flush_output(ok_result.is_ready) File "/usr/local/lib/python3.9/site-packages/pika/adapters/blocking_connection.py", line 1339, in _flush_output raise self._closing_reason # pylint: disable=E0702

[SamuelHassine]

Hello @skrumzy,

Can you please try the following procedure:

• Stop the connector
• Delete the connector in the OpenCTI UI
• Restart workers

And double check if the ID of the connector you've deleted is the same ID that is in your error message : pushe866a073-a2ce-44d6-867e-5203738aa30d (the ID of the connector is after the "push".

Kind regards, Samuel

[Kalkran]

This is happening for us too on our dev environment for enrichment connectors that are listed in the connectors page but haven't actually connected to the platform after recreating the platform. The above three steps did resolve the issue, but it seems like an oversight where the queues don't (yet) exist but workers are trying to connect to it. Maybe this could be solved by creating missing queues in rabbitmq on platform startup?

The logs generated by these errors (x3 workers) exceeded 20G in three days and filled the remaining storage.