search

ansible-middleware / amq

A collection to manage AMQ brokers
Apache License 2.0
16 stars 11 forks source link

Make amq-broker.service optionally dependent on NFS mount in unit file #35

Closed RobertFloor closed 1 year ago

RobertFloor commented 1 year ago
SUMMARY

We destroy our Azure VM in the evening an rebuild it in the morning. The AMQ Broker starts faster then the nfs mount we use. Therefore the broker does not become HA successfully since it got a lock on the directory before it was mounted by nfs. This results in two active brokers. A solution could be the approach described here making the amq-broker service dependent on the nfs mount.: https://unix.stackexchange.com/questions/246935/set-systemd-service-to-execute-after-fstab-mount. For that we would need to add an additional variable in the systemd template. Would it be possible to implement this in the code? 

# {{ ansible_managed }}
[Unit]
Description={{ activemq.instance_name }} {{ activemq.name }} Service
After=network.target **data-amq\x2dbroker-shared.mount** 

[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/{{ activemq.instance_name }}
PIDFile={{ activemq.instance_home }}/{{ activemq_service_pidfile }}
ExecStart={{ activemq.instance_home }}/bin/artemis-service start
ExecStop={{ activemq.instance_home }}/bin/artemis-service stop
SuccessExitStatus = 0 143
RestartSec = 60
Restart = on-failure
LimitNOFILE=102642

[Install]
WantedBy=multi-user.target
ISSUE TYPE
ANSIBLE VERSION
❯ ansible --version
ansible [core 2.13.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/robert/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/linuxbrew/.linuxbrew/Cellar/ansible/6.6.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /home/robert/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/linuxbrew/.linuxbrew/bin/ansible
  python version = 3.10.9 (main, Dec  6 2022, 18:44:57) [GCC 11.3.0]
  jinja version = 3.1.2
  libyaml = True
COLLECTION VERSION

# /home/robert/.ansible/collections/ansible_collections
Collection                                Version
----------------------------------------- -------
ansible.posix                             1.4.0
community.general                         6.0.1
middleware_automation.amq                 1.1.0
middleware_automation.redhat_csp_download 1.2.2

# /home/linuxbrew/.linuxbrew/Cellar/ansible/6.6.0/libexec/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    3.5.0
ansible.netcommon             3.1.3
ansible.posix                 1.4.0
ansible.utils                 2.7.0
ansible.windows               1.12.0
arista.eos                    5.0.1
awx.awx                       21.8.0
azure.azcollection            1.14.0
check_point.mgmt              2.3.0
chocolatey.chocolatey         1.3.1
cisco.aci                     2.3.0
cisco.asa                     3.1.0
cisco.dnac                    6.6.0
cisco.intersight              1.0.20
cisco.ios                     3.3.2
cisco.iosxr                   3.3.1
cisco.ise                     2.5.8
cisco.meraki                  2.11.0
cisco.mso                     2.1.0
cisco.nso                     1.0.3
cisco.nxos                    3.2.0
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.2
community.aws                 3.6.0
community.azure               1.1.0
community.ciscosmb            1.0.5
community.crypto              2.8.1
community.digitalocean        1.22.0
community.dns                 2.4.0
community.docker              2.7.1
community.fortios             1.0.0
community.general             5.8.0
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         3.4.0
community.hrobot              1.6.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             4.0.1
community.okd                 2.2.0
community.postgresql          2.3.0
community.proxysql            1.4.0
community.rabbitmq            1.2.3
community.routeros            2.3.1
community.sap                 1.0.0
community.sap_libs            1.3.0
community.skydive             1.0.0
community.sops                1.4.1
community.vmware              2.10.1
community.windows             1.11.1
community.zabbix              1.8.0
containers.podman             1.9.4
cyberark.conjur               1.2.0
cyberark.pas                  1.0.14
dellemc.enterprise_sonic      1.1.2
dellemc.openmanage            5.5.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.20.0
fortinet.fortimanager         2.1.6
fortinet.fortios              2.1.7
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.8.2
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.10.0
infinidat.infinibox           1.3.7
infoblox.nios_modules         1.4.0
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         3.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.0.4
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.21.0
netapp.elementsw              21.7.0
netapp.ontap                  21.24.1
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.3.1
netbox.netbox                 3.8.1
ngine_io.cloudstack           2.2.4
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.2
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.3.1
purestorage.flasharray        1.14.0
purestorage.flashblade        1.10.0
purestorage.fusion            1.1.1
sensu.sensu_go                1.13.1
servicenow.servicenow         1.0.6
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.31.4
theforeman.foreman            3.7.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.3.0
vyos.vyos                     3.0.1
wti.remote                    1.0.4
STEPS TO REPRODUCE
all:
  children:
    amq:
      children:
        ha1:
          hosts: xxxxxx
        ha2:
          hosts: xxxxxx  
      vars:
        activemq_configure_firewalld: True
        activemq_prometheus_enabled: True
        activemq_cors_strict_checking: False
        activemq_ha_enabled: true
        activemq_shared_storage: true
        activemq_shared_storage_path: /data/amq-broker/shared
        ansible_user: xxxx
        ansible_ssh_private_key_file: hostfiles/privkey
        activemq_offline_install: True
        activemq_version: 7.10.2
        activemq_dest: /opt/amq
        activemq_archive: "amq-broker-{{ activemq_version }}-bin.zip"
        activemq_installdir: "{{ activemq_dest }}/amq-broker-{{ activemq_version }}"
        activemq_port: 61616
        activemq_acceptors:
          - name: amqp
            bind_address: "0.0.0.0"
            bind_port: "{{ activemq_port }}"
            parameters:
              tcpSendBufferSize: 1048576
              tcpReceiveBufferSize: 1048576
              protocols: AMQP
              useEpoll: true
              amqpMinLargeMessageSize: 102400
              amqpCredits: 1000
              amqpLowCredits: 300
              amqpDuplicateDetection: true
EXPECTED RESULTS

One of the two brokers should become active the other one should remain passive. Therefore the amq-broker should start after the NFS mount.

ACTUAL RESULTS
Both brokers are active. If we restart both brokers using systemctl then only one broker becomes active.
guidograzioli commented 1 year ago

Very well reported as usual; I've not been able to reproduce the scenario in a molecule test on containers (yet...), but still releasing the bugfix as soon as it is merged. Can I ask the nfs4 mount parameters you have in use?

ivoschiphol commented 1 year ago

Hi,

My name is Ivo, im also a member of same team Robert is working for.

Here are our nfs4 mount parameters.


  -  name: Mount NFS volume
  -  become: true
  -  ansible.posix.mount:
     -  src:  amqtest2.file.core.windows.net:/amqtest2/amqdata 
     -  path: /data/amq-broker/shared 
     -  opts: rw,sync,hard,intr
     -  state: mounted
     -  fstype: nfs