search

Azure / azure-service-bus

☁️ Azure Service Bus service issue tracking and samples
https://azure.microsoft.com/services/service-bus
MIT License
585 stars 781 forks source link

Service Bus FailoverProvider is having issues to recover #723

Open andrianov17 opened 2 weeks ago

andrianov17 commented 2 weeks ago

From time to time, Service Bus Java client cannot recover keeping reporting that connection has been restored while it is not. And it keeps doing it forever. In result, both JMS producers and consumers are effectively dead.

Java library used: com.azure.spring:spring-cloud-azure-starter-servicebus-jms:4.17.0.

Below are symptom log messages, repeating every minute (real namespace is replaced with [namespace]): 2024-11-02T00:06:02.745-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:07:02.945-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:08:03.177-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:09:03.425-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:10:03.662-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:11:03.903-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:12:04.083-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:13:04.342-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:14:04.522-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:15:04.672-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:16:04.858-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:17:05.064-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:18:05.268-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:19:05.39-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net ...

EldertGrootenboer commented 2 weeks ago

We have added an investigation task for this, we will update this thread once we have more details.