search

ritsec / cluster-duck

Documentation for the RITSEC Cloud.
6 stars 2 forks source link

Nova nodes are displaying client unexpectedly closed TCP connection in rabbitmq logs #35

Closed poptar7 closed 4 years ago

poptar7 commented 4 years ago

Most Nova nodes are reporting being down

Most Nova nodes are reporting being down, Nova05 is the only node currently reporting being up.

The service on the nodes is still running(active), with errors.

poptar7 commented 4 years ago

The errors being shown in the logs is

ERROR oslo_service.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID

and

WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 50.01 sec

poptar7 commented 4 years ago

The Node that is marked "up" has the same errors as the down nodes.

poptar7 commented 4 years ago

Nova client is giving "AMQP server on 10.0.10.66:5671 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: IOError: Server unexpectedly closed connection"

10.0.10.66 is on mgmt01