search

spring-projects / spring-boot

Spring Boot
https://spring.io/projects/spring-boot
Apache License 2.0
74.43k stars 40.52k forks source link

Spring Boot Micro Service Not Defined in Registry When JMS Server Not Reachable #22628

Closed Christophe06410 closed 4 years ago

Christophe06410 commented 4 years ago

I've already created a Stack overflow question for this issue, but I have no reply so far. See: here

I have a strange issue that took me several days to narrow down. Basically, I have a Jhipster project based on Spring boot Version 2.1.10.RELEASE, which contains 4 microservices. We are interested here in 2 of them: Gateway and Corehub. In the gateway, I have an angular app that performs a POST to /services/corehub/api/someendpoint which used to be working and that is failing now with different error messages, but the one I have more regularly is 

{
  "type": "https://www.jhipster.tech/problem/problem-with-message",
  "title": "Method Not Allowed",
  "status": 405,
  "detail": "Request method 'POST' not supported",
  "path": "/services/ambientcorehub/api/trips",
  "message": "error.http.405"
}

I ended up looking at the traces of the Registry that keeps track of the microservices for internal communication and I found out that when this error occurs, I cannot find the corehub in the traces anymore. So it looks like the corehub micro service is not registered.

An other GIT branch of this service does not have this problem, so I performed a diff between these two branches and I removed the changes until I could narrow down the problem. So, in the corehub, I have a JMS listener based on this mq-jms-spring implementation. The maven dependency is as follows:

        <dependency>
            <groupId>com.ibm.mq</groupId>
            <artifactId>mq-jms-spring-boot-starter</artifactId>
            <version>2.2.7</version>
        </dependency>

I commented out my JmsListener class and its associated JmsContext (to get access to a topic), and kept only the configuration properties defining access to the server, along with the port, channel, topic name, etc.

If I comment the above maven dependency, my service works again. If I keep the maven dependency, with the configuration only, my corehub microservice is not registered in the Registry and becomes not accessible anymore from the gateway and thus the Angular UI.

What is important to note, is that I have currently some network issue which prevents me from accessing the JMS Server. So I believe the exception that is raised by this IBM library because the JMS server is not reachable, breaks the registration of the microservice towards the spring boot registry.

Here are the traces that come over and over in the corehub console:

2020-07-22 08:21:45.316  WARN 7964 --- [nfoReplicator-0] o.s.boot.actuate.jms.JmsHealthIndicator  : JMS health check failed

com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name '172.31.14.1(9010)'.
    at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:489)
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
    at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:448)
    at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
    at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
    at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:303)
    at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
    at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6005)
    at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6030)
    at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:409)
    at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:349)
    at org.springframework.jms.connection.SingleConnectionFactory.getConnection(SingleConnectionFactory.java:327)
    at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:242)
    at org.springframework.boot.actuate.jms.JmsHealthIndicator.doHealthCheck(JmsHealthIndicator.java:52)
    at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95)
    at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:110)
    at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:106)
    at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1406)
    at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
    ... 24 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '172.31.14.1(9010)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]],3=172.31.14.1(9010),5=RemoteTCPConnection.bindAndConnectSocket]
    at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13558)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1426)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1385)
    at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
    at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
    at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:381)
    ... 23 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:901)
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1381)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:976)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:553)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:233)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:141)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13302)
    ... 28 common frames omitted
Caused by: java.net.ConnectException: Connection timed out: connect
    at java.base/java.net.PlainSocketImpl.connect0(Native Method)
    at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at java.base/java.net.Socket.connect(Socket.java:558)
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1022)
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1014)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connectSocket(RemoteTCPConnection.java:1014)
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:805)
    ... 35 common frames omitted

Here are some version numbers:

Here is my configuration in application.yml:

spring:
  jms:
    # Used for JMS Message reception.
    isPubSubDomain: false

application:
  oag:
    # This can be a queue or a topic (if subdomain is defined)
    # In case of a topic, sub domain must be set to public.
    queueName: "BRIDGE.XXX.TO.YYY.TST"
    isTopic: false

ibm:
  mq:
    queueManager:
    channel: XXX_GWT11.BT1
    connName: 172.31.14.1(9010)
    user: xxxx
    password:

Would it be possible to catch this exception to avoid breaking regular Spring Boot registration mechanism? I cannot afford having my cluster down because I cannot access the JMS server.

Beside this, I opened this message on IBM MQ side here and a person suggested me to stop the JMS health indicator. So I set the following property to no avail:

management:
  endpoint:
    jms:
      # Prevent Unreachable JMS Server from unregistering corehub from the registry, leading to unreachable microservice from the Gateway
      enabled: false

Corresponding documentation is here

Any help would be greatly appreciated. Thank you

wilkinsona commented 4 years ago

In the interests of keeping the discussion in one place, I'm going to close this in favour of your question on Stack Overflow to which I have just posted an answer.