Unable to create clustered brokers

garethahealy commented 6 months ago

SUMMARY

If amq_broker_ha_enabled is enabled, the broker.xml generates connectors and cluster-connections which do not allow for message flow between nodes.

ISSUE TYPE

Bug Report

ANSIBLE VERSION

$ ansible --version
ansible [core 2.15.9]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ec2-user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ec2-user/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/ec2-user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ec2-user/.local/bin/ansible
  python version = 3.9.18 (main, Sep  7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

COLLECTION VERSION

$  ansible-galaxy collection list

# /home/ec2-user/.ansible/collections/ansible_collections
Collection                    Version
----------------------------- -------
redhat.amq_broker             2.0.0
redhat.runtimes_common        1.1.3

STEPS TO REPRODUCE

deploy-amq.yml

---
- name: Playbook to deploy activemq in clustered
  hosts: all
  vars:
    amq_broker_ha_enabled: true
    amq_broker_nio_enabled: true
    amq_broker_cors_strict_checking: false
    amq_broker_prometheus_enabled: false
    amq_broker_offline_install: true
    amq_broker_archive: amq-broker-7.11.6-bin.zip
    amq_broker_addresses:
      - name: TEST
        anycast:
        - name: TEST
      - name: DLQ
        anycast:
        - name: DLQ
      - name: ExpiryQueue
        anycast:
          - name: ExpiryQueue
    amq_broker_acceptors:
      - name: artemis
        bind_address: "0.0.0.0"
        bind_port: "{{ amq_broker_port }}"
        parameters:
          tcpSendBufferSize: 1048576
          tcpReceiveBufferSize: 1048576
          amqpMinLargeMessageSize: 102400
          protocols: CORE
          useEpoll: true
          amqpCredits: 1000
          amqpLowCredits: 300
          amqpDuplicateDetection: true
          supportAdvisory: false
          suppressInternalManagementObjects: false
      - name: amqp
        bind_address: "0.0.0.0"
        bind_port: "{{ amq_broker_port_amqp }}"
        parameters:
          tcpSendBufferSize: 1048576
          tcpReceiveBufferSize: 1048576
          protocols: AMQP
          useEpoll: true
          amqpMinLargeMessageSize: 102400
          amqpCredits: 1000
          amqpLowCredits: 300
          amqpDuplicateDetection: true
  roles:
    - redhat.amq_broker.amq_broker

inventory/group_vars/all.yml

---
all:
  hosts:
    ip-11-0-1-001.eu-west-2.compute.internal:
      amq_broker_instance_name: one
    ip-11-0-1-002.eu-west-2.compute.internal:
      amq_broker_instance_name: two
    ip-11-0-1-003.eu-west-2.compute.internal:
      amq_broker_instance_name: three

Create 3 EC2 instances
Once brokers deployed
Attach consumer to broker two and three, amq/bin/artemis consumer --user amq-broker --password amq-broker --message-count 500
Attach producer to broker one, amq/bin/artemis producer --user amq-broker --password amq-broker
Nothing happens.

EXPECTED RESULTS

Broker two and three get 500 messages each.

Ansible created samples, which don't allow for clustered broker:

<connectors>
  <connector name="artemis">tcp://localhost:61616?</connector>
  <connector name="ip-11-0-1-001.eu-west-2.compute.internal">tcp://11.0.1.001:61616?</connector>
  <connector name="ip-11-0-1-002.eu-west-2.compute.internal">tcp://11.0.1.002:61616?</connector>
  <connector name="ip-11-0-1-003.eu-west-2.compute.internal">tcp://11.0.1.003:61616?</connector>
</connectors>

<cluster-connections>
  <cluster-connection name="amq_broker">
    <connector-ref>artemis</connector-ref>
    <message-load-balancing>ON_DEMAND</message-load-balancing>
    <max-hops>1</max-hops>
    <static-connectors>
      <connector-ref>ip-11-0-1-001.eu-west-2.compute.internal</connector-ref>
      <connector-ref>ip-11-0-1-002.eu-west-2.compute.internal</connector-ref>
      <connector-ref>ip-11-0-1-003.eu-west-2.compute.internal</connector-ref>
    </static-connectors>
  </cluster-connection>
</cluster-connections>

Manually edited broker one sample.

Changed artemis from localhost to actual IP
Removed ip-11-0-1-001.eu-west-2.compute.internal from static-connectors

<connectors>
  <connector name="artemis">tcp://11.0.1.001:61616?</connector>
  <connector name="ip-11-0-1-002.eu-west-2.compute.internal">tcp://11.0.1.002:61616?</connector>
  <connector name="ip-11-0-1-003.eu-west-2.compute.internal">tcp://11.0.1.003:61616?</connector>
</connectors>

<cluster-connections>
  <cluster-connection name="amq_broker">
    <connector-ref>artemis</connector-ref>
    <message-load-balancing>ON_DEMAND</message-load-balancing>
    <max-hops>1</max-hops>
    <static-connectors>
      <connector-ref>ip-11-0-1-002.eu-west-2.compute.internal</connector-ref>
      <connector-ref>ip-11-0-1-003.eu-west-2.compute.internal</connector-ref>
    </static-connectors>
  </cluster-connection>
</cluster-connections>

The above config works, and broker two and three get 500 messages.

garethahealy commented 6 months ago

@guidograzioli i dont think i am setting anything wrong in the vars.

Looking at the code, the cluster-connections always expects to be localhost:

https://github.com/ansible-middleware/amq/blob/main/roles/activemq/templates/cluster_connections.broker.xml.j2#L2

And the connectors just loops over whats provided:

https://github.com/ansible-middleware/amq/blob/main/roles/activemq/templates/connectors.broker.xml.j2

which in my case, is the ansible inventory:

https://github.com/ansible-middleware/amq/blob/main/roles/activemq/tasks/main.yml#L7-L26

which means there will always be 3 items in the static-connectors, where as there should be only 2.

guidograzioli commented 6 months ago

I'll try to reproduce, however, unless something changed with the most recent version of artemis that was just updated, having three static-connectors for two node seems the correct configuration, which you can verify by running:

apache-artemis-2.32.0/bin/artemis create --clustered --staticCluster="tcp://server1:61616,tcp://server2:61616" test

which should produce:

      <connectors>
            <connector name="artemis">tcp://localhost:61616</connector>
            <connector name = "node0">tcp://server1:61616</connector>
            <connector name = "node1">tcp://server2:61616</connector>
      </connectors>

      <cluster-connections>
         <cluster-connection name="my-cluster">
            <connector-ref>artemis</connector-ref>
            <message-load-balancing>ON_DEMAND</message-load-balancing>
            <max-hops>0</max-hops>
            <static-connectors>
               <connector-ref>node0</connector-ref>
               <connector-ref>node1</connector-ref>
            </static-connectors>
         </cluster-connection>
      </cluster-connections>

guidograzioli commented 4 weeks ago

Closing as not reproducible, welcome to reopen if you face the issue again.

garethahealy commented 3 weeks ago

Still re-producible 100% of the time.

Deploy: https://github.com/garethahealy/ansible-middleware-playground/blob/main/deploy-amq-clustered.yml Test: https://github.com/garethahealy/ansible-middleware-playground/blob/main/test-amq-clustered.yml

ansible-middleware / amq