Support for multiple queues and queue managers

[Jeff-Walker]

I'm looking at mq-jms-spring-boot-starter:2.0.0. Is there any support for having connections to multiple queue managers?

I'm guessing I can't have autoconfiguration happen for me?

[ibmmqmet]

There's nothing explicit in the configuration code to allow anything like multiple sets of connections from a single properties file. I guess you might be able to use "default" configuration values that could apply to all queue managers and then override specific fields in a customizer method. And there's probably mechanisms in Spring itself to pass different sets of resource values when creating a CF but it's not something I've tried. It's not about MQ, but this link may give some hints on dealing with multiple CFs.

[Jeff-Walker]

What I ended up doing for now is making a @Bean method that produces a list of MQConnectionFactorys. I introduced my own properties class that uses yours.

@ConfigurationProperties("mq")
@Configuration
public class MqProperties {
  private List<MQConfigurationProperties> servers;

  public List<MQConfigurationProperties> getServers() {
    return servers;
  }
  public void setServers(List<MQConfigurationProperties> servers) {
    this.servers = servers;
  }
}

My application.yml file:

mq:
  servers: 
    - queue-manager: qm1
      channel: chan
      conn-name: host(port)
      user: usr
      password: passwd
    - queue-manager: qm2
      channel: chan
      conn-name: host2(port)
      user: usr
      password: passwd   

 @Bean
  public List<MQConnectionFactory> factories() throws JMSException {
    List<MQConnectionFactory> factories = new ArrayList<>();
    for (MQConfigurationProperties server : properties.getServers()) {
      MQConnectionFactory cf = new MQConnectionFactory();
      String qmName = server.getQueueManager();
      cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, qmName);
   ...

I have to duplicate your MQConnectionFactoryFactory code because it's package protected, which I totally understand, it's a rational design choice.

The queue managers have the same type of data, just load-balanced between the two servers. I haven't used IBM MQ in 20 years (back when it was MQ Series), but when using the WebSphere default provider, I would have had a cluster level SIB to present a single endpoint to the consumer, but I'm not the middleware guy. I also haven't done JMS in Spring before, but I'm assuming I'll end up with two instances of JmsTemplate, which again must short-circuit Spring's auto configuration.

But, anyway, I'm not sure how you would provide this type of functionality without breaking changes, but your project was very helpful to me in wiring up my own factories.

Thanks.

[abedwardsw]

not 100% sure, but I think if you use client channel definition files you can achieve load balancing from the client. I belive mq-jms-spring would need to be updated, but might be a better solution. https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q032510_.htm http://www-01.ibm.com/support/docview.wss?uid=swg21508357

[harshalkh]

Can we use MQConnectionFactoryCustomizer for connection through client channel definition files CCDT, without updating the mq-jms-spring starter?

[jmcwho]

If MQConnectionFactoryFactory was modified to a public class with a public constructor, then multiple connections could easily be handle by defining a set of properties/MQConnectionFactory/JmsListenerContainerFactories.

Addition improvements might be able to be made to hide more details of the creation of the MQConnectionFactory (including XA versus non XA)

Here is a code snippet in case its helpful to anyone.

@Bean
@ConfigurationProperties("queue1")
public MQConfigurationProperties queue1MQConfigProperties() {
    return new MQConfigurationProperties();
}

@Bean
public MQConnectionFactory queue1ConnectionFactory(@Qualifier("queue1MQConfigProperties") MQConfigurationProperties properties, ObjectProvider<List<MQConnectionFactoryCustomizer>> factoryCustomizers)
{
    return new MQConnectionFactoryFactory(properties, (List)factoryCustomizers.getIfAvailable()).createConnectionFactory(MQConnectionFactory.class);
}

@Bean
public JmsListenerContainerFactory<?> queue1MsgFactory(@Qualifier("queue1ConnectionFactory") ConnectionFactory connectionFactory, DefaultJmsListenerContainerFactoryConfigurer configurer) {
    DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();

    // This provides all boot's defaults to the this factory, including the message converter
    configurer.configure(factory, connectionFactory);

    // You could still override some of Boot's defaults if necessary.
    factory.setMessageConverter(simpleMsgConverter());

    return factory;
}

[clejk59]

What I ended up doing for now is making a @Bean method that produces a list of MQConnectionFactorys. I introduced my own properties class that uses yours.

@ConfigurationProperties("mq")
@Configuration
public class MqProperties {
  private List<MQConfigurationProperties> servers;

  public List<MQConfigurationProperties> getServers() {
    return servers;
  }
  public void setServers(List<MQConfigurationProperties> servers) {
    this.servers = servers;
  }
}

My application.yml file:

mq:
  servers: 
    - queue-manager: qm1
      channel: chan
      conn-name: host(port)
      user: usr
      password: passwd
    - queue-manager: qm2
      channel: chan
      conn-name: host2(port)
      user: usr
      password: passwd   

 @Bean
  public List<MQConnectionFactory> factories() throws JMSException {
    List<MQConnectionFactory> factories = new ArrayList<>();
    for (MQConfigurationProperties server : properties.getServers()) {
      MQConnectionFactory cf = new MQConnectionFactory();
      String qmName = server.getQueueManager();
      cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, qmName);
   ...

I have to duplicate your MQConnectionFactoryFactory code because it's package protected, which I totally understand, it's a rational design choice.

The queue managers have the same type of data, just load-balanced between the two servers. I haven't used IBM MQ in 20 years (back when it was MQ Series), but when using the WebSphere default provider, I would have had a cluster level SIB to present a single endpoint to the consumer, but I'm not the middleware guy. I also haven't done JMS in Spring before, but I'm assuming I'll end up with two instances of JmsTemplate, which again must short-circuit Spring's auto configuration.

But, anyway, I'm not sure how you would provide this type of functionality without breaking changes, but your project was very helpful to me in wiring up my own factories.

Thanks.

Does your approach support Same queue defined in multiple Queue managers, host and channels ? Can you share your github link? trying to figure out the rest of the methods cachingConnectionFactory and jmsTransactionMgr, jmsOperations & listenerContainerFactory. did you have a looping on each of this method ?

[Jeff-Walker]

In my approach, i define totally separate connection factories, so I'm not sure why it wouldn't work.

I can't give you a repo link, as the code is proprietary, but my final result was like this:

  @Bean
  public List<QueueInfo> jmsTemplates() throws JMSException {
    return
        properties.getMq().getServersWithDefaults()
        .stream()
        .map(server -> {
          MQConnectionFactory connectionFactory = createConnectionFactory(server);
          JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory);
          jmsTemplate.setReceiveTimeout(server.getReadTimeout().toMillis());
          jmsTemplate.setDefaultDestinationName(server.getQueueName());
          jmsTemplate.setMessageConverter(messageConverter);
          return new QueueInfo(server.getQueueManager(), jmsTemplate);
        })
        .collect(Collectors.toList());
  }

  protected static MQConnectionFactory createConnectionFactory(MqServer server) {
    try {
      MQConnectionFactory cf = new MQConnectionFactory();
      cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, server.getQueueManager());

      cf.setStringProperty(WMQConstants.WMQ_CONNECTION_NAME_LIST, server.getConnectionName());
      cf.setStringProperty(WMQConstants.WMQ_CHANNEL, server.getChannel());
      cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);

      cf.setStringProperty(WMQConstants.USERID, server.getUser());
      cf.setStringProperty(WMQConstants.PASSWORD, server.getPassword());
      cf.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, false);

      return cf;
    } catch (JMSException j) {
      throw JmsUtils.convertJmsAccessException(j);
    }
  }

The QueueInfo is a value object that has a name and the JmsOperations. Then to use it, i autowire the list of QueueInfos. For this app I was using Spring Batch, so I looped over these objects and defined JmsItemReaders for parallel steps.

Hope this helps. I never tried any of the suggestions after what I posted, as my project played out and I haven't looked at it again since.

[Deeepthi143]

clejk59 : Hi , i am new to Queues Concept. i have same queue name defined in multiple Queue managers, host and channels?how can i call them dynamically to Listen to the queue with out duplicate the code. as of now i am using two DefaultJmsListenerContainerFactory to connect to the different host and queuemanger , and also using two @JmsListener to receive messages from queue. i want to write the generic code to connect to multiple host and QueueMangers with multiple queues.

[oscarsan]

I used the same aproach of @Jeff-Walker but combined a bit with the solution in https://stackoverflow.com/questions/43399072/spring-boot-configure-multiple-activemq-instances, so to not use any other external class. It works perfectly, in this case just for 2 queues, and can be tested with the docker https://github.com/ibm-messaging/mq-docker, by making to container with different outside ports.

Off course, this no need some configuration but is a good starting point if you don't want to make those new classes and objects.

For creating 2 dockers just make

docker run --env LICENSE=accept --env MQ_QMGR_NAME=QM1 --publish 1414:1414 --publish 9443:9443 --detach ibmcom/mq

docker run --env LICENSE=accept --env MQ_QMGR_NAME=QM2 --publish 1415:1414 --publish 9444:9443 --detach ibmcom/mq

@Configuration
public class JmsConfig {
    public static final String LOCAL_Q = "localQ";
    public static final String REMOTE_Q = "remoteQ";

    @Bean
    @Primary
    public ConnectionFactory jmsConnectionFactory() {
        MQConnectionFactory connectionFactory = createConnectionFactory();
        return connectionFactory;
    }

    @Bean
    public ConnectionFactory jmsConnectionFactory2() {
        MQConnectionFactory connectionFactory = createConnectionFactory2();
        return connectionFactory;
    }

    protected static MQConnectionFactory createConnectionFactory() {
        try {
            MQConnectionFactory cf = new MQConnectionFactory();
            cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, "QM1");

            cf.setStringProperty(WMQConstants.WMQ_CONNECTION_NAME_LIST, "localhost(1414)");
            cf.setStringProperty(WMQConstants.WMQ_CHANNEL, "DEV.ADMIN.SVRCONN");
            cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);

            cf.setStringProperty(WMQConstants.USERID, "admin");
            cf.setStringProperty(WMQConstants.PASSWORD, "passw0rd");
            cf.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, false);

            return cf;
        } catch (JMSException j) {
            throw JmsUtils.convertJmsAccessException(j);
        }
    }

    protected static MQConnectionFactory createConnectionFactory2() {
        try {
            MQConnectionFactory cf = new MQConnectionFactory();
            cf.setStringProperty(WMQConstants.WMQ_QUEUE_MANAGER, "QM2");

            cf.setStringProperty(WMQConstants.WMQ_CONNECTION_NAME_LIST, "localhost(1415)");
            cf.setStringProperty(WMQConstants.WMQ_CHANNEL, "DEV.ADMIN.SVRCONN");
            cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);

            cf.setStringProperty(WMQConstants.USERID, "admin");
            cf.setStringProperty(WMQConstants.PASSWORD, "passw0rd");
            cf.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, false);

            return cf;
        } catch (JMSException j) {
            throw JmsUtils.convertJmsAccessException(j);
        }
    }

    @Bean
    @Primary
    public JmsTemplate jmsTemplate() {
        JmsTemplate jmsTemplate = new JmsTemplate();
        jmsTemplate.setConnectionFactory(jmsConnectionFactory());
        jmsTemplate.setDefaultDestinationName(LOCAL_Q);
        return jmsTemplate;
    }

    @Bean
    public JmsTemplate jmsTemplate2() {
        JmsTemplate jmsTemplate = new JmsTemplate();
        jmsTemplate.setConnectionFactory(jmsConnectionFactory2());
        jmsTemplate.setDefaultDestinationName(REMOTE_Q);
        return jmsTemplate;
    }

    @Bean
    @Primary
    public JmsListenerContainerFactory<?> queueFactoryConfig(ConnectionFactory connectionFactory,
            DefaultJmsListenerContainerFactoryConfigurer configurer) {
        DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
        configurer.configure(factory, connectionFactory);
        factory.setPubSubDomain(false);
        return factory;
    }

    @Bean
    @Primary
    public JmsListenerContainerFactory<?> queueFactoryConfig2(@Qualifier("jmsConnectionFactory2") ConnectionFactory connectionFactory,
            DefaultJmsListenerContainerFactoryConfigurer configurer) {
        DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
        configurer.configure(factory, connectionFactory);
        factory.setPubSubDomain(false);
        return factory;
    }

[adzubla]

As an example, I've made a runnable demo. It is described at https://dev.to/adzubla/using-multiple-jms-servers-with-spring-boot-3cbm

[jacquesvdm7]

This is great work

[cemcanoglu]

My new work will help you. https://github.com/cemcanoglu