Closed jeanetteclark closed 5 months ago
from metadig-rabbitmq-2
on metadig-prod
2024-01-27 01:17:15.503584+00:00 [info] <0.29211.57> Node rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local is down, deleting its listeners
2024-01-27 01:17:15.506133+00:00 [info] <0.29211.57> 1 transient queues from an old incarnation of node 'rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local' deleted in 0.002372s
2024-01-27 01:17:15.506828+00:00 [info] <0.29211.57> node 'rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local' down: net_tick_timeout
2024-01-27 01:17:15.508839+00:00 [error] <0.29211.57> Partial partition detected:
2024-01-27 01:17:15.508839+00:00 [error] <0.29211.57> * We saw DOWN from rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local
2024-01-27 01:17:15.508839+00:00 [error] <0.29211.57> * We can still see rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local which can see rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local
2024-01-27 01:17:15.508839+00:00 [error] <0.29211.57> We will therefore intentionally disconnect from rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local
2024-01-27 01:17:16.509849+00:00 [info] <0.29211.57> rabbit on node 'rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local' down
2024-01-27 01:17:16.689184+00:00 [info] <0.29211.57> Node rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local is down, deleting its listeners
2024-01-27 01:17:16.690188+00:00 [info] <0.29211.57> node 'rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local' down: disconnect
2024-01-27 01:17:17.564922+00:00 [warning] <0.5490.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:17.564922+00:00 [warning] <0.5490.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:17.564922+00:00 [warning] <0.5490.71>
2024-01-27 01:17:23.707971+00:00 [info] <0.29211.57> node 'rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local' up
2024-01-27 01:17:23.709438+00:00 [error] <0.28697.57> Mnesia('rabbit@metadig-rabbitmq-2.metadig-rabbitmq-headless.metadig.svc.cluster.local'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local'}
2024-01-27 01:17:23.709438+00:00 [error] <0.28697.57>
2024-01-27 01:17:23.711015+00:00 [warning] <0.5504.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:23.711015+00:00 [warning] <0.5504.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:23.711015+00:00 [warning] <0.5504.71>
2024-01-27 01:17:23.900701+00:00 [warning] <0.5512.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:23.900701+00:00 [warning] <0.5512.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:23.900701+00:00 [warning] <0.5512.71>
2024-01-27 01:17:26.163949+00:00 [error] <0.29652.57> Failed to fetch a list of nodes from Kubernetes API: timeout
2024-01-27 01:17:26.165908+00:00 [warning] <0.5520.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:26.165908+00:00 [warning] <0.5520.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:26.165908+00:00 [warning] <0.5520.71>
2024-01-27 01:17:28.427964+00:00 [warning] <0.29652.57> Peer discovery: node rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local is unreachable
2024-01-27 01:17:28.428091+00:00 [warning] <0.29652.57> Peer discovery: node rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local is unreachable
2024-01-27 01:17:30.913963+00:00 [warning] <0.5515.71> global: 'rabbit@metadig-rabbitmq-2.metadig-rabbitmq-headless.metadig.svc.cluster.local' failed to connect to 'rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local'
2024-01-27 01:17:30.913963+00:00 [warning] <0.5515.71>
2024-01-27 01:17:33.901482+00:00 [warning] <0.5533.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:33.901482+00:00 [warning] <0.5533.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:33.901482+00:00 [warning] <0.5533.71>
2024-01-27 01:17:35.984562+00:00 [info] <0.29211.57> node 'rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local' up
2024-01-27 01:17:35.988391+00:00 [warning] <0.5537.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:35.988391+00:00 [warning] <0.5537.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:35.988391+00:00 [warning] <0.5537.71>
2024-01-27 01:17:36.165953+00:00 [error] <0.29652.57> Failed to fetch a list of nodes from Kubernetes API: timeout
2024-01-27 01:17:36.167540+00:00 [warning] <0.5543.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:36.167540+00:00 [warning] <0.5543.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:36.167540+00:00 [warning] <0.5543.71>
2024-01-27 01:17:38.430971+00:00 [warning] <0.29652.57> Peer discovery: node rabbit@metadig-rabbitmq-0.metadig-rabbitmq-headless.metadig.svc.cluster.local is unreachable
2024-01-27 01:17:38.431106+00:00 [warning] <0.29652.57> Peer discovery: node rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local is unreachable
2024-01-27 01:17:43.901316+00:00 [warning] <0.5568.71> Description: "Authenticity is not established by certificate path validation"
2024-01-27 01:17:43.901316+00:00 [warning] <0.5568.71> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2024-01-27 01:17:43.901316+00:00 [warning] <0.5568.71>
2024-01-27 01:17:44.324627+00:00 [warning] <0.29211.57> Autoheal: we were selected to restart; winner is 'rabbit@metadig-rabbitmq-1.metadig-rabbitmq-headless.metadig.svc.cluster.local'
2024-01-27 01:17:44.324726+00:00 [info] <0.5576.71> RabbitMQ is asked to stop...
2024-01-27 01:17:44.364111+00:00 [info] <0.5576.71> Stopping RabbitMQ applications and their dependencies in the following order:
in an effort to fix the above, I upgraded rabbitmq on dev-metadig
and have been putting it through its paces by sending the test.arctic node a whole slew of documents (around 400 in total). I have seen zero connection errors so far. the process for upgrading rabbitmq was:
helm upgrade
to chart version 11.10.1helm upgrade
to chart version 12.8.2edit: nevermind see a few dropped connections now
updated rabbitmq with deployment of 3.0.0 and so far so good...
Awesome.
RabbitMQ is set up to auto-recover connections that fail for network reasons, but sometimes this auto-recovery seems to fail.
Here are the logs showing the connection reset by peer error, an apparent channel recovery, then the exception
NB: The application is currently running the bitnami chart 10.3.9 for rabbitmq which has the 3.10.8 release of rabbitmq. this release reached EOL on Dec 31 so we should probably upgrade rabbitmq to the latest version which is 12.8.0
Edit: more log messages from RMQ that might be important