search

LINBIT / drbd-utils

DRBD userspace utilities (for 9.x, 8.4, 8.3)
GNU General Public License v2.0
78 stars 46 forks source link

crm-unfence-peer.9.sh "cannot parse input" when trying to unfence resource #11

Closed SpitchAG closed 2 years ago

SpitchAG commented 2 years ago

DRBDADM_BUILDTAG=GIT-hash:\ 9f2aa37c2e0438bc3e6336eb8f697cfc6c0b8243\ build\ by\ root@ip-172-31-12-231.eu-west-1.compute.internal\,\ 2021-11-14\ 18:45:36 DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION_CODE=0x090104 DRBD_KERNEL_VERSION=9.1.4 DRBDADM_VERSION_CODE=0x091300 DRBDADM_VERSION=9.19.0

when a resource location constraint has been placed for a resource on a node, if at some point drbd try to unfence the resource, it cannot , and drbd resource cannot be promoted anymore on the node ,

error seems to be the call to sudo cibadmin --modify --allow-create -o constraints -X "" (called from create_or_modify_constraint) (note the empty data "" at the end). Indeed the new_constraint value in the bash script does not seem to be ever set,

rck commented 2 years ago

that looks like something for @lge

lge commented 2 years ago

"Does not work" is not enough to go on.

SpitchAG commented 2 years ago

DRBD config resource audiodata { startup { wfc-timeout 15; degr-wfc-timeout 60; }

    volume 0 {
            device /dev/drbd0;
            disk /dev/mapper/audio_pool-audio_volume;
            meta-disk internal;
    }

    on ip-172-31-12-83 {
            node-id   0;
            address 172.31.12.83:7788;
    }
    on ip-172-31-12-98 {
            node-id   1;
            address 172.31.12.98:7788;
    }

    connection {
         host ip-172-31-12-83    address 172.31.12.83:7788;
         host ip-172-31-12-98    address 172.31.12.98:7788;
         net {
            protocol C;
            cram-hmac-alg sha1;
            shared-secret "secret";
            after-sb-0pri discard-younger-primary;
            after-sb-1pri discard-secondary;
            after-sb-2pri disconnect;
                                fencing resource-and-stonith;
                         }
    }

            # begin geo cluster extension
    on ip-172-31-12-208 {
         node-id   2;
         address 172.31.12.208:7788;
    }
    on ip-172-31-12-57 {
            node-id   3;
            address 172.31.12.57:7788;
    }
    # extended local cluster synchronous links
    connection {
         host ip-172-31-12-208   address 172.31.12.208:7788;
         host ip-172-31-12-57   address 172.31.12.57:7788;
         net {
            protocol C;
            cram-hmac-alg sha1;
            shared-secret "secret";
            after-sb-0pri discard-younger-primary;
            after-sb-1pri discard-secondary;
            after-sb-2pri disconnect;
                                fencing resource-and-stonith;
                         }
    }

    #async links between local clusters
    connection {
         host ip-172-31-12-83       address 172.31.12.83:7788;
         host ip-172-31-12-208   address 172.31.12.208:7788;
         net {
             protocol A;
         }
    }
    connection {
         host ip-172-31-12-83       address 172.31.12.83:7788;
         host ip-172-31-12-57   address 172.31.12.57:7788;
         net {
             protocol A;
         }
    }
    connection {
         host ip-172-31-12-98       address 172.31.12.98:7788;
         host ip-172-31-12-208   address 172.31.12.208:7788;
         net {
             protocol A;
         }
    }
    connection {
         host ip-172-31-12-98       address 172.31.12.98:7788;
         host ip-172-31-12-57   address 172.31.12.57:7788;
         net {
             protocol A;
         }
    }
    # end geo cluster extension

    handlers {
                  fence-peer "/usr/local/lib/drbd/crm-fence-peer.9.sh";
        after-resync-target "/usr/local/lib/drbd/crm-unfence-peer.9.sh";
              }

}

* cib when DRBD is "healthy", Cluster Name: spitch-storage-cluster Corosync Nodes: ip-172-31-12-83 ip-172-31-12-98 Pacemaker Nodes: ip-172-31-12-83 ip-172-31-12-98

Resources: Clone: health-aws-clone Resource: health-aws (class=ocf provider=spitch type=HealthAWS) Meta Attrs: failure-timeout=60s Operations: monitor interval=20 (health-aws-monitor-interval-20) start interval=0s timeout=10 (health-aws-start-interval-0s) stop interval=0s on-fail=ignore (health-aws-stop-interval-0s) Clone: net-snmp-clone Resource: net-snmp (class=ocf provider=spitch type=anything) Attributes: binfile=/usr/local/spitch/net-snmp/sbin/snmpd cmdline_options="-f -Lo" logfile=/var/opt/spitch/net-snmp/log/net-snmp.log Operations: monitor interval=5 (net-snmp-monitor-interval-5) start interval=0s timeout=20s (net-snmp-start-interval-0s) stop interval=0s timeout=20s (net-snmp-stop-interval-0s) Clone: stats-monitor-clone Resource: stats-monitor (class=ocf provider=spitch type=anything) Attributes: binfile=/opt/spitch/stats-monitor/bin/stats-monitor cmdline_options="-s -r -c /opt/spitch/stats-monitor/conf/certs/rest-agent.crt -C /opt/spitch/stats-monitor/conf/certs/ca.pem -k /opt/spitch/stats-monitor/conf/certs/rest-agent.key" Operations: monitor interval=5 (stats-monitor-monitor-interval-5) start interval=0s timeout=20s (stats-monitor-start-interval-0s) stop interval=0s timeout=20s (stats-monitor-stop-interval-0s) Clone: gw-ping-clone Resource: gw-ping (class=ocf provider=pacemaker type=ping) Attributes: host_list=172.31.12.1 Operations: monitor interval=20 (gw-ping-monitor-interval-20) start interval=0s timeout=60 (gw-ping-start-interval-0s) stop interval=0s timeout=20 (gw-ping-stop-interval-0s) Master: audiodata-clone Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true Resource: audiodata (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=audiodata drbdconf=/usr/local/etc/drbd.conf Operations: demote interval=0s timeout=90 (audiodata-demote-interval-0s) monitor interval=30 (audiodata-monitor-interval-30) notify interval=0s timeout=90 (audiodata-notify-interval-0s) promote interval=0s timeout=90 (audiodata-promote-interval-0s) reload interval=0s timeout=30 (audiodata-reload-interval-0s) start interval=0s timeout=240 (audiodata-start-interval-0s) stop interval=0s timeout=100 (audiodata-stop-interval-0s) Clone: audio-lvm-clone Resource: audio-lvm (class=ocf provider=heartbeat type=LVM) Attributes: volgrpname=audio_pool Operations: methods interval=0s timeout=5s (audio-lvm-methods-interval-0s) monitor interval=30 (audio-lvm-monitor-interval-30) start interval=0s timeout=30s (audio-lvm-start-interval-0s) stop interval=0s timeout=30s (audio-lvm-stop-interval-0s) Resource: audio-fs (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/drbd0 directory=/mnt/audio fstype=xfs Operations: monitor interval=30 (audio-fs-monitor-interval-30) notify interval=0s timeout=60s (audio-fs-notify-interval-0s) start interval=0s timeout=60s (audio-fs-start-interval-0s) stop interval=0s timeout=60s (audio-fs-stop-interval-0s) Group: audio-vip-grp Meta Attrs: resource-stickiness=500 Resource: audio-vip (class=ocf provider=heartbeat type=IPaddr2) Attributes: cidr_netmask=24 ip=172.31.12.162 Operations: monitor interval=30 (audio-vip-monitor-interval-30) start interval=0s timeout=20s (audio-vip-start-interval-0s) stop interval=0s timeout=20s (audio-vip-stop-interval-0s) Resource: audio-awsalias (class=ocf provider=heartbeat type=awsvip) Attributes: secondary_private_ip=172.31.12.162 Meta Attrs: failure-timeout=60s Operations: migrate_from interval=0s timeout=30s (audio-awsalias-migrate_from-interval-0s) migrate_to interval=0s timeout=30s (audio-awsalias-migrate_to-interval-0s) monitor interval=30 (audio-awsalias-monitor-interval-30) start interval=0s timeout=30s (audio-awsalias-start-interval-0s) stop interval=0s on-fail=ignore (audio-awsalias-stop-interval-0s) validate interval=0s timeout=10s (audio-awsalias-validate-interval-0s) Group: audio-restgw-grp Meta Attrs: resource-stickiness=500 Resource: audio-cleanup (class=ocf provider=spitch type=SingleRunScript) Attributes: options=/mnt/audio script=/opt/spitch/storage/bin/storage_cleanup.sh Operations: monitor interval=30 (audio-cleanup-monitor-interval-30) start interval=0s timeout=30 (audio-cleanup-start-interval-0s) stop interval=0s timeout=30 (audio-cleanup-stop-interval-0s) Resource: audio-nginx (class=ocf provider=heartbeat type=nginx) Attributes: configfile=/opt/spitch/storage/rest-gw/nginx/conf/nginx.audio.conf httpd=/usr/local/openresty/nginx/sbin/nginx Operations: monitor interval=30 (audio-nginx-monitor-interval-30) reload interval=0s timeout=40s (audio-nginx-reload-interval-0s) start interval=0s on-fail=fence (audio-nginx-start-interval-0s) stop interval=0s timeout=60s (audio-nginx-stop-interval-0s) Master: voiceprintsdata-clone Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true Resource: voiceprintsdata (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=voiceprintsdata drbdconf=/usr/local/etc/drbd.conf Operations: demote interval=0s timeout=90 (voiceprintsdata-demote-interval-0s) monitor interval=30 (voiceprintsdata-monitor-interval-30) notify interval=0s timeout=90 (voiceprintsdata-notify-interval-0s) promote interval=0s timeout=90 (voiceprintsdata-promote-interval-0s) reload interval=0s timeout=30 (voiceprintsdata-reload-interval-0s) start interval=0s timeout=240 (voiceprintsdata-start-interval-0s) stop interval=0s timeout=100 (voiceprintsdata-stop-interval-0s) Clone: voiceprints-lvm-clone Resource: voiceprints-lvm (class=ocf provider=heartbeat type=LVM) Attributes: volgrpname=voiceprints_pool Operations: methods interval=0s timeout=5s (voiceprints-lvm-methods-interval-0s) monitor interval=30 (voiceprints-lvm-monitor-interval-30) start interval=0s timeout=30s (voiceprints-lvm-start-interval-0s) stop interval=0s timeout=30s (voiceprints-lvm-stop-interval-0s) Resource: voiceprints-fs (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/drbd1 directory=/mnt/voiceprints fstype=xfs Operations: monitor interval=30 (voiceprints-fs-monitor-interval-30) notify interval=0s timeout=60s (voiceprints-fs-notify-interval-0s) start interval=0s timeout=60s (voiceprints-fs-start-interval-0s) stop interval=0s timeout=60s (voiceprints-fs-stop-interval-0s) Group: voiceprints-vip-grp Meta Attrs: resource-stickiness=500 Resource: voiceprints-vip (class=ocf provider=heartbeat type=IPaddr2) Attributes: cidr_netmask=24 ip=172.31.12.13 Operations: monitor interval=30 (voiceprints-vip-monitor-interval-30) start interval=0s timeout=20s (voiceprints-vip-start-interval-0s) stop interval=0s timeout=20s (voiceprints-vip-stop-interval-0s) Resource: voiceprints-awsalias (class=ocf provider=heartbeat type=awsvip) Attributes: secondary_private_ip=172.31.12.13 Meta Attrs: failure-timeout=60s Operations: migrate_from interval=0s timeout=30s (voiceprints-awsalias-migrate_from-interval-0s) migrate_to interval=0s timeout=30s (voiceprints-awsalias-migrate_to-interval-0s) monitor interval=30 (voiceprints-awsalias-monitor-interval-30) start interval=0s timeout=30s (voiceprints-awsalias-start-interval-0s) stop interval=0s on-fail=ignore (voiceprints-awsalias-stop-interval-0s) validate interval=0s timeout=10s (voiceprints-awsalias-validate-interval-0s) Group: voiceprints-restgw-grp Meta Attrs: resource-stickiness=500 Resource: voiceprints-cleanup (class=ocf provider=spitch type=SingleRunScript) Attributes: options=/mnt/voiceprints script=/opt/spitch/storage/bin/storage_cleanup.sh Operations: monitor interval=30 (voiceprints-cleanup-monitor-interval-30) start interval=0s timeout=30 (voiceprints-cleanup-start-interval-0s) stop interval=0s timeout=30 (voiceprints-cleanup-stop-interval-0s) Resource: voiceprints-nginx (class=ocf provider=heartbeat type=nginx) Attributes: configfile=/opt/spitch/storage/rest-gw/nginx/conf/nginx.voiceprints.conf httpd=/usr/local/openresty/nginx/sbin/nginx Operations: monitor interval=30 (voiceprints-nginx-monitor-interval-30) reload interval=0s timeout=40s (voiceprints-nginx-reload-interval-0s) start interval=0s on-fail=fence (voiceprints-nginx-start-interval-0s) stop interval=0s timeout=60s (voiceprints-nginx-stop-interval-0s) Group: booth-booth-group Resource: booth-booth-ip (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=172.31.12.145 Operations: monitor interval=10s timeout=20s (booth-booth-ip-monitor-interval-10s) start interval=0s timeout=20s (booth-booth-ip-start-interval-0s) stop interval=0s timeout=20s (booth-booth-ip-stop-interval-0s) Resource: booth-booth-service (class=ocf provider=pacemaker type=booth-site) Attributes: config=/etc/booth/booth.conf Operations: monitor interval=10 start-delay=0 timeout=20 (booth-booth-service-monitor-interval-10) reload interval=0s timeout=20 (booth-booth-service-reload-interval-0s) restart interval=0s timeout=20 (booth-booth-service-restart-interval-0s) start interval=0s timeout=20 (booth-booth-service-start-interval-0s) stop interval=0s timeout=20 (booth-booth-service-stop-interval-0s) Resource: booth-ip-alias (class=ocf provider=heartbeat type=awsvip) Attributes: secondary_private_ip=172.31.12.145 Meta Attrs: failure-timeout=60s Operations: migrate_from interval=0s timeout=30s (booth-ip-alias-migrate_from-interval-0s) migrate_to interval=0s timeout=30s (booth-ip-alias-migrate_to-interval-0s) monitor interval=20s timeout=30s (booth-ip-alias-monitor-interval-20s) start interval=0s timeout=30s (booth-ip-alias-start-interval-0s) stop interval=0s on-fail=ignore (booth-ip-alias-stop-interval-0s) validate interval=0s timeout=10s (booth-ip-alias-validate-interval-0s)

Stonith Devices: Resource: fence-ec2 (class=stonith type=fence_aws) Attributes: pcmk_host_map=ip-172-31-12-83:i-0345cec975f9668c4;ip-172-31-12-98:i-035c6499a0672f691 power_wait=180 Operations: monitor interval=60s (fence-ec2-monitor-interval-60s) Resource: fence-sbd (class=stonith type=fence_sbd) Attributes: devices=/dev/sdf power_timeout=30 Operations: monitor interval=60s (fence-sbd-monitor-interval-60s) Fencing Levels: Target: . Level 1 - fence-ec2 Level 2 - fence-sbd

Location Constraints: Resource: audio-restgw-grp Constraint: location-audio-restgw-grp-ping Rule: boolean-op=or score=-INFINITY (id:location-audio-restgw-grp-ping-rule) Expression: pingd lt 1 (id:location-audio-restgw-grp-ping-rule-expr) Expression: not_defined pingd (id:location-audio-restgw-grp-ping-rule-expr-1) Resource: fence-ec2 Enabled on: ip-172-31-12-83 (score:500) (id:location-fence-ec2-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-fence-ec2-ip-172-31-12-98-500) Resource: fence-sbd Enabled on: ip-172-31-12-83 (score:500) (id:location-fence-sbd-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-fence-sbd-ip-172-31-12-98-500) Resource: health-aws-clone Enabled on: ip-172-31-12-83 (score:600) (id:location-health-aws-ip-172-31-12-83-600) Enabled on: ip-172-31-12-98 (score:600) (id:location-health-aws-ip-172-31-12-98-600) Resource: net-snmp-clone Enabled on: ip-172-31-12-83 (score:500) (id:location-net-snmp-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-net-snmp-ip-172-31-12-98-500) Resource: stats-monitor-clone Enabled on: ip-172-31-12-83 (score:500) (id:location-stats-monitor-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-stats-monitor-ip-172-31-12-98-500) Resource: voiceprints-restgw-grp Constraint: location-voiceprints-restgw-grp-ping Rule: boolean-op=or score=-INFINITY (id:location-voiceprints-restgw-grp-ping-rule) Expression: pingd lt 1 (id:location-voiceprints-restgw-grp-ping-rule-expr) Expression: not_defined pingd (id:location-voiceprints-restgw-grp-ping-rule-expr-1) Ordering Constraints: start audio-lvm-clone then start audiodata-clone (kind:Mandatory) (id:order-audio-lvm-clone-audiodata-clone-mandatory) promote audiodata-clone then start audio-fs (kind:Mandatory) (id:order-audiodata-clone-audio-fs-mandatory) start audiodata-clone then start audio-fs (kind:Mandatory) (id:order-audiodata-clone-audio-fs-mandatory-1) start audio-fs then start audio-restgw-grp (kind:Mandatory) (id:order-audio-fs-audio-restgw-grp-mandatory) start voiceprints-lvm-clone then start voiceprintsdata-clone (kind:Mandatory) (id:order-voiceprints-lvm-clone-voiceprintsdata-clone-mandatory) promote voiceprintsdata-clone then start voiceprints-fs (kind:Mandatory) (id:order-voiceprintsdata-clone-voiceprints-fs-mandatory) start voiceprintsdata-clone then start voiceprints-fs (kind:Mandatory) (id:order-voiceprintsdata-clone-voiceprints-fs-mandatory-1) start voiceprints-fs then start voiceprints-restgw-grp (kind:Mandatory) (id:order-voiceprints-fs-voiceprints-restgw-grp-mandatory) Colocation Constraints: audio-fs with audiodata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-audio-fs-audiodata-clone-INFINITY) audio-vip-grp with audiodata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-audio-vip-grp-audiodata-clone-INFINITY) audio-restgw-grp with audio-fs (score:INFINITY) (id:colocation-audio-restgw-grp-audio-fs-INFINITY) voiceprints-fs with voiceprintsdata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-voiceprints-fs-voiceprintsdata-clone-INFINITY) voiceprints-vip-grp with voiceprintsdata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-voiceprints-vip-grp-voiceprintsdata-clone-INFINITY) voiceprints-restgw-grp with voiceprints-fs (score:INFINITY) (id:colocation-voiceprints-restgw-grp-voiceprints-fs-INFINITY) booth-ip-alias with booth-booth-group (score:INFINITY) (id:colocation-booth-ip-alias-booth-booth-group-INFINITY) Ticket Constraints: Master audiodata-clone loss-policy=demote ticket=ticket-storagedata (id:ticket-ticket-storagedata-audiodata-clone-Master) Master voiceprintsdata-clone loss-policy=demote ticket=ticket-storagedata (id:ticket-ticket-storagedata-voiceprintsdata-clone-Master)

Alerts: Alert: snmp_alert (path=/var/lib/pacemaker/alert_snmp.sh) Description: snmp_alert pacemaker alert Options: trap_binary=/usr/local/spitch/net-snmp/bin/snmptrap trap_community=spitch trap_options=-Ci -M /opt/spitch/ha-misc/monitor:/usr/local/spitch/net-snmp/share/snmp/mibs -u spitch -l authPriv -a MD5 -A XXX -x DES -X YYY trap_snmp_persistent_dir=/var/opt/spitch/net-snmp trap_version=3 Meta options: timestamp-format=%Y-%m-%d,%H:%M:%S.%01N Recipients: Recipient: snmp_alert-recipient (value=172.31.1.73)

Resources Defaults: No defaults set Operations Defaults: No defaults set

Cluster Properties: cluster-infrastructure: corosync cluster-name: spitch-storage-cluster dc-version: 1.1.23-1.el7_9.1-9acf116022 have-watchdog: true no-quorum-policy: ignore node-health-red: -550 node-health-strategy: progressive stonith-action: off stonith-enabled: True stonith-timeout: 240 symmetric-cluster: true

Quorum: Options:

Resources: Clone: health-aws-clone Resource: health-aws (class=ocf provider=spitch type=HealthAWS) Meta Attrs: failure-timeout=60s Operations: monitor interval=20 (health-aws-monitor-interval-20) start interval=0s timeout=10 (health-aws-start-interval-0s) stop interval=0s on-fail=ignore (health-aws-stop-interval-0s) Clone: net-snmp-clone Resource: net-snmp (class=ocf provider=spitch type=anything) Attributes: binfile=/usr/local/spitch/net-snmp/sbin/snmpd cmdline_options="-f -Lo" logfile=/var/opt/spitch/net-snmp/log/net-snmp.log Operations: monitor interval=5 (net-snmp-monitor-interval-5) start interval=0s timeout=20s (net-snmp-start-interval-0s) stop interval=0s timeout=20s (net-snmp-stop-interval-0s) Clone: stats-monitor-clone Resource: stats-monitor (class=ocf provider=spitch type=anything) Attributes: binfile=/opt/spitch/stats-monitor/bin/stats-monitor cmdline_options="-s -r -c /opt/spitch/stats-monitor/conf/certs/rest-agent.crt -C /opt/spitch/stats-monitor/conf/certs/ca.pem -k /opt/spitch/stats-monitor/conf/certs/rest-agent.key" Operations: monitor interval=5 (stats-monitor-monitor-interval-5) start interval=0s timeout=20s (stats-monitor-start-interval-0s) stop interval=0s timeout=20s (stats-monitor-stop-interval-0s) Clone: gw-ping-clone Resource: gw-ping (class=ocf provider=pacemaker type=ping) Attributes: host_list=172.31.12.1 Operations: monitor interval=20 (gw-ping-monitor-interval-20) start interval=0s timeout=60 (gw-ping-start-interval-0s) stop interval=0s timeout=20 (gw-ping-stop-interval-0s) Master: audiodata-clone Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true Resource: audiodata (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=audiodata drbdconf=/usr/local/etc/drbd.conf Operations: demote interval=0s timeout=90 (audiodata-demote-interval-0s) monitor interval=30 (audiodata-monitor-interval-30) notify interval=0s timeout=90 (audiodata-notify-interval-0s) promote interval=0s timeout=90 (audiodata-promote-interval-0s) reload interval=0s timeout=30 (audiodata-reload-interval-0s) start interval=0s timeout=240 (audiodata-start-interval-0s) stop interval=0s timeout=100 (audiodata-stop-interval-0s) Clone: audio-lvm-clone Resource: audio-lvm (class=ocf provider=heartbeat type=LVM) Attributes: volgrpname=audio_pool Operations: methods interval=0s timeout=5s (audio-lvm-methods-interval-0s) monitor interval=30 (audio-lvm-monitor-interval-30) start interval=0s timeout=30s (audio-lvm-start-interval-0s) stop interval=0s timeout=30s (audio-lvm-stop-interval-0s) Resource: audio-fs (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/drbd0 directory=/mnt/audio fstype=xfs Operations: monitor interval=30 (audio-fs-monitor-interval-30) notify interval=0s timeout=60s (audio-fs-notify-interval-0s) start interval=0s timeout=60s (audio-fs-start-interval-0s) stop interval=0s timeout=60s (audio-fs-stop-interval-0s) Group: audio-vip-grp Meta Attrs: resource-stickiness=500 Resource: audio-vip (class=ocf provider=heartbeat type=IPaddr2) Attributes: cidr_netmask=24 ip=172.31.12.162 Operations: monitor interval=30 (audio-vip-monitor-interval-30) start interval=0s timeout=20s (audio-vip-start-interval-0s) stop interval=0s timeout=20s (audio-vip-stop-interval-0s) Resource: audio-awsalias (class=ocf provider=heartbeat type=awsvip) Attributes: secondary_private_ip=172.31.12.162 Meta Attrs: failure-timeout=60s Operations: migrate_from interval=0s timeout=30s (audio-awsalias-migrate_from-interval-0s) migrate_to interval=0s timeout=30s (audio-awsalias-migrate_to-interval-0s) monitor interval=30 (audio-awsalias-monitor-interval-30) start interval=0s timeout=30s (audio-awsalias-start-interval-0s) stop interval=0s on-fail=ignore (audio-awsalias-stop-interval-0s) validate interval=0s timeout=10s (audio-awsalias-validate-interval-0s) Group: audio-restgw-grp Meta Attrs: resource-stickiness=500 Resource: audio-cleanup (class=ocf provider=spitch type=SingleRunScript) Attributes: options=/mnt/audio script=/opt/spitch/storage/bin/storage_cleanup.sh Operations: monitor interval=30 (audio-cleanup-monitor-interval-30) start interval=0s timeout=30 (audio-cleanup-start-interval-0s) stop interval=0s timeout=30 (audio-cleanup-stop-interval-0s) Resource: audio-nginx (class=ocf provider=heartbeat type=nginx) Attributes: configfile=/opt/spitch/storage/rest-gw/nginx/conf/nginx.audio.conf httpd=/usr/local/openresty/nginx/sbin/nginx Operations: monitor interval=30 (audio-nginx-monitor-interval-30) reload interval=0s timeout=40s (audio-nginx-reload-interval-0s) start interval=0s on-fail=fence (audio-nginx-start-interval-0s) stop interval=0s timeout=60s (audio-nginx-stop-interval-0s) Master: voiceprintsdata-clone Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true Resource: voiceprintsdata (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=voiceprintsdata drbdconf=/usr/local/etc/drbd.conf Operations: demote interval=0s timeout=90 (voiceprintsdata-demote-interval-0s) monitor interval=30 (voiceprintsdata-monitor-interval-30) notify interval=0s timeout=90 (voiceprintsdata-notify-interval-0s) promote interval=0s timeout=90 (voiceprintsdata-promote-interval-0s) reload interval=0s timeout=30 (voiceprintsdata-reload-interval-0s) start interval=0s timeout=240 (voiceprintsdata-start-interval-0s) stop interval=0s timeout=100 (voiceprintsdata-stop-interval-0s) Clone: voiceprints-lvm-clone Resource: voiceprints-lvm (class=ocf provider=heartbeat type=LVM) Attributes: volgrpname=voiceprints_pool Operations: methods interval=0s timeout=5s (voiceprints-lvm-methods-interval-0s) monitor interval=30 (voiceprints-lvm-monitor-interval-30) start interval=0s timeout=30s (voiceprints-lvm-start-interval-0s) stop interval=0s timeout=30s (voiceprints-lvm-stop-interval-0s) Resource: voiceprints-fs (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/drbd1 directory=/mnt/voiceprints fstype=xfs Operations: monitor interval=30 (voiceprints-fs-monitor-interval-30) notify interval=0s timeout=60s (voiceprints-fs-notify-interval-0s) start interval=0s timeout=60s (voiceprints-fs-start-interval-0s) stop interval=0s timeout=60s (voiceprints-fs-stop-interval-0s) Group: voiceprints-vip-grp Meta Attrs: resource-stickiness=500 Resource: voiceprints-vip (class=ocf provider=heartbeat type=IPaddr2) Attributes: cidr_netmask=24 ip=172.31.12.13 Operations: monitor interval=30 (voiceprints-vip-monitor-interval-30) start interval=0s timeout=20s (voiceprints-vip-start-interval-0s) stop interval=0s timeout=20s (voiceprints-vip-stop-interval-0s) Resource: voiceprints-awsalias (class=ocf provider=heartbeat type=awsvip) Attributes: secondary_private_ip=172.31.12.13 Meta Attrs: failure-timeout=60s Operations: migrate_from interval=0s timeout=30s (voiceprints-awsalias-migrate_from-interval-0s) migrate_to interval=0s timeout=30s (voiceprints-awsalias-migrate_to-interval-0s) monitor interval=30 (voiceprints-awsalias-monitor-interval-30) start interval=0s timeout=30s (voiceprints-awsalias-start-interval-0s) stop interval=0s on-fail=ignore (voiceprints-awsalias-stop-interval-0s) validate interval=0s timeout=10s (voiceprints-awsalias-validate-interval-0s) Group: voiceprints-restgw-grp Meta Attrs: resource-stickiness=500 Resource: voiceprints-cleanup (class=ocf provider=spitch type=SingleRunScript) Attributes: options=/mnt/voiceprints script=/opt/spitch/storage/bin/storage_cleanup.sh Operations: monitor interval=30 (voiceprints-cleanup-monitor-interval-30) start interval=0s timeout=30 (voiceprints-cleanup-start-interval-0s) stop interval=0s timeout=30 (voiceprints-cleanup-stop-interval-0s) Resource: voiceprints-nginx (class=ocf provider=heartbeat type=nginx) Attributes: configfile=/opt/spitch/storage/rest-gw/nginx/conf/nginx.voiceprints.conf httpd=/usr/local/openresty/nginx/sbin/nginx Operations: monitor interval=30 (voiceprints-nginx-monitor-interval-30) reload interval=0s timeout=40s (voiceprints-nginx-reload-interval-0s) start interval=0s on-fail=fence (voiceprints-nginx-start-interval-0s) stop interval=0s timeout=60s (voiceprints-nginx-stop-interval-0s) Group: booth-booth-group Resource: booth-booth-ip (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=172.31.12.145 Operations: monitor interval=10s timeout=20s (booth-booth-ip-monitor-interval-10s) start interval=0s timeout=20s (booth-booth-ip-start-interval-0s) stop interval=0s timeout=20s (booth-booth-ip-stop-interval-0s) Resource: booth-booth-service (class=ocf provider=pacemaker type=booth-site) Attributes: config=/etc/booth/booth.conf Operations: monitor interval=10 start-delay=0 timeout=20 (booth-booth-service-monitor-interval-10) reload interval=0s timeout=20 (booth-booth-service-reload-interval-0s) restart interval=0s timeout=20 (booth-booth-service-restart-interval-0s) start interval=0s timeout=20 (booth-booth-service-start-interval-0s) stop interval=0s timeout=20 (booth-booth-service-stop-interval-0s) Resource: booth-ip-alias (class=ocf provider=heartbeat type=awsvip) Attributes: secondary_private_ip=172.31.12.145 Meta Attrs: failure-timeout=60s Operations: migrate_from interval=0s timeout=30s (booth-ip-alias-migrate_from-interval-0s) migrate_to interval=0s timeout=30s (booth-ip-alias-migrate_to-interval-0s) monitor interval=20s timeout=30s (booth-ip-alias-monitor-interval-20s) start interval=0s timeout=30s (booth-ip-alias-start-interval-0s) stop interval=0s on-fail=ignore (booth-ip-alias-stop-interval-0s) validate interval=0s timeout=10s (booth-ip-alias-validate-interval-0s)

Stonith Devices: Resource: fence-ec2 (class=stonith type=fence_aws) Attributes: pcmk_host_map=ip-172-31-12-83:i-0345cec975f9668c4;ip-172-31-12-98:i-035c6499a0672f691 power_wait=180 Operations: monitor interval=60s (fence-ec2-monitor-interval-60s) Resource: fence-sbd (class=stonith type=fence_sbd) Attributes: devices=/dev/sdf power_timeout=30 Operations: monitor interval=60s (fence-sbd-monitor-interval-60s) Fencing Levels: Target: . Level 1 - fence-ec2 Level 2 - fence-sbd

Location Constraints: Resource: audio-restgw-grp Constraint: location-audio-restgw-grp-ping Rule: boolean-op=or score=-INFINITY (id:location-audio-restgw-grp-ping-rule) Expression: pingd lt 1 (id:location-audio-restgw-grp-ping-rule-expr) Expression: not_defined pingd (id:location-audio-restgw-grp-ping-rule-expr-1) Resource: audiodata-clone Constraint: drbd-fence-by-handler-audiodata-audiodata-clone Rule: role=Master score=-INFINITY (id:drbd-fence-by-handler-audiodata-rule-audiodata-clone) Expression: #uname ne ip-172-31-12-98 (id:drbd-fence-by-handler-audiodata-expr-1-audiodata-clone) Expression: #uname ne ip-172-31-12-208 (id:drbd-fence-by-handler-audiodata-expr-2-audiodata-clone) Expression: #uname ne ip-172-31-12-57 (id:drbd-fence-by-handler-audiodata-expr-3-audiodata-clone) Resource: fence-ec2 Enabled on: ip-172-31-12-83 (score:500) (id:location-fence-ec2-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-fence-ec2-ip-172-31-12-98-500) Resource: fence-sbd Enabled on: ip-172-31-12-83 (score:500) (id:location-fence-sbd-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-fence-sbd-ip-172-31-12-98-500) Resource: health-aws-clone Enabled on: ip-172-31-12-83 (score:600) (id:location-health-aws-ip-172-31-12-83-600) Enabled on: ip-172-31-12-98 (score:600) (id:location-health-aws-ip-172-31-12-98-600) Resource: net-snmp-clone Enabled on: ip-172-31-12-83 (score:500) (id:location-net-snmp-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-net-snmp-ip-172-31-12-98-500) Resource: stats-monitor-clone Enabled on: ip-172-31-12-83 (score:500) (id:location-stats-monitor-ip-172-31-12-83-500) Enabled on: ip-172-31-12-98 (score:500) (id:location-stats-monitor-ip-172-31-12-98-500) Resource: voiceprints-restgw-grp Constraint: location-voiceprints-restgw-grp-ping Rule: boolean-op=or score=-INFINITY (id:location-voiceprints-restgw-grp-ping-rule) Expression: pingd lt 1 (id:location-voiceprints-restgw-grp-ping-rule-expr) Expression: not_defined pingd (id:location-voiceprints-restgw-grp-ping-rule-expr-1) Ordering Constraints: start audio-lvm-clone then start audiodata-clone (kind:Mandatory) (id:order-audio-lvm-clone-audiodata-clone-mandatory) promote audiodata-clone then start audio-fs (kind:Mandatory) (id:order-audiodata-clone-audio-fs-mandatory) start audiodata-clone then start audio-fs (kind:Mandatory) (id:order-audiodata-clone-audio-fs-mandatory-1) start audio-fs then start audio-restgw-grp (kind:Mandatory) (id:order-audio-fs-audio-restgw-grp-mandatory) start voiceprints-lvm-clone then start voiceprintsdata-clone (kind:Mandatory) (id:order-voiceprints-lvm-clone-voiceprintsdata-clone-mandatory) promote voiceprintsdata-clone then start voiceprints-fs (kind:Mandatory) (id:order-voiceprintsdata-clone-voiceprints-fs-mandatory) start voiceprintsdata-clone then start voiceprints-fs (kind:Mandatory) (id:order-voiceprintsdata-clone-voiceprints-fs-mandatory-1) start voiceprints-fs then start voiceprints-restgw-grp (kind:Mandatory) (id:order-voiceprints-fs-voiceprints-restgw-grp-mandatory) Colocation Constraints: audio-fs with audiodata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-audio-fs-audiodata-clone-INFINITY) audio-vip-grp with audiodata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-audio-vip-grp-audiodata-clone-INFINITY) audio-restgw-grp with audio-fs (score:INFINITY) (id:colocation-audio-restgw-grp-audio-fs-INFINITY) voiceprints-fs with voiceprintsdata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-voiceprints-fs-voiceprintsdata-clone-INFINITY) voiceprints-vip-grp with voiceprintsdata-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-voiceprints-vip-grp-voiceprintsdata-clone-INFINITY) voiceprints-restgw-grp with voiceprints-fs (score:INFINITY) (id:colocation-voiceprints-restgw-grp-voiceprints-fs-INFINITY) booth-ip-alias with booth-booth-group (score:INFINITY) (id:colocation-booth-ip-alias-booth-booth-group-INFINITY) Ticket Constraints: Master audiodata-clone loss-policy=demote ticket=ticket-storagedata (id:ticket-ticket-storagedata-audiodata-clone-Master) Master voiceprintsdata-clone loss-policy=demote ticket=ticket-storagedata (id:ticket-ticket-storagedata-voiceprintsdata-clone-Master)

Alerts: Alert: snmp_alert (path=/var/lib/pacemaker/alert_snmp.sh) Description: snmp_alert pacemaker alert Options: trap_binary=/usr/local/spitch/net-snmp/bin/snmptrap trap_community=spitch trap_options=-Ci -M /opt/spitch/ha-misc/monitor:/usr/local/spitch/net-snmp/share/snmp/mibs -u spitch -l authPriv -a MD5 -A SpitchAG -x DES -X SpitchAG trap_snmp_persistent_dir=/var/opt/spitch/net-snmp trap_version=3 Meta options: timestamp-format=%Y-%m-%d,%H:%M:%S.%01N Recipients: Recipient: snmp_alert-recipient (value=172.31.1.73)

Resources Defaults: No defaults set Operations Defaults: No defaults set

Cluster Properties: cluster-infrastructure: corosync cluster-name: spitch-storage-cluster dc-version: 1.1.23-1.el7_9.1-9acf116022 have-watchdog: true no-quorum-policy: ignore node-health-red: -550 node-health-strategy: progressive stonith-action: off stonith-enabled: True stonith-timeout: 240 symmetric-cluster: true

Quorum: Options:

lge commented 2 years ago

"Works for me".