rabbitmqctl status returned non-zero

[matelakat]

This happened on 22 Apr, on node d00-25-b5-a0-00-ff rabbit's monitor reported failure. This triggered the stop and restart of rabbit, which led to some neutron agents being thought to be offline, thus migrated.

pacemaker logs:

Apr 22 10:55:47 [4629] d00-25-b5-a0-00-ff       crmd:     info: process_lrm_event:      Operation rabbitmq_monitor_10000: not running (node=d00-25-b
5-a0-00-ff, call=624, rc=7, cib-update=824, confirmed=false)
Apr 22 10:55:47 [4624] d00-25-b5-a0-00-ff        cib:     info: cib_process_request:    Forwarding cib_modify operation for section status to master
 (origin=local/crmd/824)
Apr 22 10:55:47 [4624] d00-25-b5-a0-00-ff        cib:     info: cib_perform_op: Diff: --- 0.1571.46 2
Apr 22 10:55:47 [4624] d00-25-b5-a0-00-ff        cib:     info: cib_perform_op: Diff: +++ 0.1571.47 (null)
...

Apr 22 10:55:47 [4626] d00-25-b5-a0-00-ff       lrmd:   notice: log_execute:    executing - rsc:rabbitmq action:stop call_id:646

rabbit logs:

=INFO REPORT==== 22-Apr-2017::10:45:18 ===
accepting AMQP connection <0.17354.49> (10.76.4.136:59532 -> 10.76.4.109:5672)
=INFO REPORT==== 22-Apr-2017::10:45:18 ===
accepting AMQP connection <0.17357.49> (10.76.4.98:51510 -> 10.76.4.109:5672)
=WARNING REPORT==== 22-Apr-2017::10:45:30 ===
closing AMQP connection <0.4450.1> (10.76.4.98:59921 -> 10.76.4.109:5672):
connection_closed_abruptly
=WARNING REPORT==== 22-Apr-2017::10:45:30 ===
closing AMQP connection <0.30782.0> (10.76.4.136:51008 -> 10.76.4.109:5672):
connection_closed_abruptly
=ERROR REPORT==== 22-Apr-2017::10:53:17 ===
** Connection attempt from disallowed node 'rabbitmq-plugins27157@d00-25-b5-a0-00-fa' **
=INFO REPORT==== 22-Apr-2017::10:55:48 ===
Stopping RabbitMQ
=INFO REPORT==== 22-Apr-2017::10:55:49 ===
stopped TCP Listener on 10.76.4.109:5672
=ERROR REPORT==== 22-Apr-2017::10:55:49 ===
Error on AMQP connection <0.4278.4> (10.76.4.105:47613 -> 10.76.4.109:5672, vhost: '/openstack', user: 'openstack', state: running), channel 0:
{amqp_error,connection_forced,
            "broker forced connection closure with reason 'shutdown'",none}
=ERROR REPORT==== 22-Apr-2017::10:55:49 ===
Error on AMQP connection <0.24511.4> (10.76.4.106:49903 -> 10.76.4.109:5672, vhost: '/openstack', user: 'openstack', state: running), channel 0:
{amqp_error,connection_forced,
            "broker forced connection closure with reason 'shutdown'",none}

[matelakat]

It seems that the command line that is executed is something along the lines of:

rabbitmqctl -n rabbit@rabbitmq-default-controller status

[matelakat]

https://github.com/rabbitmq/rabbitmq-server-release/pull/26 And a refactor: https://github.com/ClusterLabs/resource-agents/pull/970

[matelakat]

I am creating an SR for the product to include the new ocf script instead of the old one.

[matelakat]

Internal SR: https://build.suse.de/request/show/132426

[matelakat]

The SR got accepted, so we need to make sure SAP is taking the packages from https://build.suse.de/package/show/Devel:Cloud:6/rabbitmq-server

[mkoderer]

@seife can you create a rpm and we test this in lab

[seife]

@matelakat can you push the package from IBS to OBS? So that I can just copypac it to our build service instance?

[rsalevsky]

@seife: https://build.opensuse.org/package/show/home:rsalevsky/rabbitmq-server

[seife]

Package is built in SAP-Cloud:SOC6:staging. @mkoderer how do I test this? And if it works, we can just take the official update from the SOC6 Update channel?

[matelakat]

@seife I think the test would look like this:

• set up a cluster, it should work - this tests if the new ocf resource script is doing the job
• shut down rabbit and see that the monitor action's output is logged by pacemaker - this is the main feature that is added by the change - if a rabbitmqctl command is returning non-zero, it should be visible in the pacemaker logs as an info message

[SebastianBiedler]

I tested the patch in the mkcloud. It seems to work

[matelakat]

Closing this issue as I believe it has been implemented (copypac 'ed) Please @seife reopen it if it's not the case.