Pika connection to RabbitMq stops working after a while

[koszta5]

Rasa version: 1.7.0 Rasa X version (if used & relevant): 0.25.2 Operating system (windows, osx, ...): Ubuntu 18.04.3 (native docker images)

This has been a re-occuring issue in RASA X (official images even). As described in multiple posts:

• it works at first, it gets broken after some time (likely continuing conversation after a long time, action server replying late..

Symptoms:

• messages sent to bot are blocked and action server is not receiving any requests. I checked rabbit mq and it is fine; up and running
• bot just does not reply

Errors look like this: WARNING rasa.core.brokers.pika - RabbitMQ channel has not been assigned. Adding message to list of unpublished messages and trying to publish them later. Current number of unpublished messages is 1. ERROR pika.adapters.utils.connection_workflow - AMQPConnector - reporting failure: AMQPConnectorAMQPHandshakeError: ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.' ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('192.168.96.6', 48136)>; error=111 (Connection refused) The following eventually shows in logs but events are still not processed: INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rabbit'.

Relevant reports of same/very similar issues: https://forum.rasa.com/t/no-more-logs-nor-interactive-learning-working/24597 https://forum.rasa.com/t/is-it-possible-to-use-rasa-x-as-a-model-server/21250/4 https://forum.rasa.com/t/rasa-x-does-not-connect-to-production-worker-hosts/24801/2 https://forum.rasa.com/t/rasa-x-rabbitmq-connection-closed-error/20556/3 https://forum.rasa.com/t/errors-in-rasa-production/23854

Current workaround:

• Do a fresh install of native rasa X docker stack

Let me know if more information is needed

[sara-tagger]

Thanks for the issue, @btotharye will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[erohmensing]

WARNING rasa.core.brokers.pika - RabbitMQ channel has not been assigned. Adding message to list of unpublished messages and trying to publish them later. Current number of unpublished messages is 1.

@alwx does this seem like something you working with Handle cases when EventService fails to add events to the database could fix?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity. Please create a new issue if you need more help.

[prakashdale]

Issue still exists with latest code.