stliu
commented
1 year ago Accoring to this doc https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-troubleshoot#connection-is-closed
an AMQP Link will be closed after 10 minutes idle
You see the following error when the AMQP connection and link are active but no calls (for example, send or receive) are made using the link for 10 minutes. So, the link is closed. The connection is still open.
amqp:link:detach-forced:The link 'G2:7223832:user.tenant0.cud_00000000000-0000-0000-0000-00000000000000' is force detached by the broker due to errors occurred in publisher(link164614). Detach origin: AmqpMessagePublisher.IdleTimerExpired: Idle timeout: 00:10:00. TrackingId:00000000000000000000000000000000000000_G2_B3, SystemTracker:mynamespace:Topic:MyTopic, Timestamp:2/16/2018 11:10:40 PM
an AMQP Connection will be closed after all links are closed
You see the following error on the AMQP connection when all links in the connection have been closed because there was no activity (idle) and a new link has not been created in 5 minutes.
Error{condition=amqp:connection:forced, description='The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:00000000000000000000000000000000000_G21, SystemTracker:gateway5, Timestamp:2019-03-06T17:32:00', info=null}
And the last one
You see this error when a new AMQP connection is created but a link is not created within 1 minute of the creation of the AMQP Connection.
Error{condition=amqp:connection:forced, description='The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:0000000000000000000000000000000000000_G21, SystemTracker:gateway5, Timestamp:2019-03-06T18:41:51', info=null}
To summary
- An AMQP Connection is a socket connection ( TCP or WebSocket ) between an application and Azure Service Bus
- An AMQP connection can have 0-n links
- A link w/o any activity of more than 10 minutes will be closed by Azure Service Bus
- A Connection w/o an active Link for more than 5 minutes ( or 1 minute for a newly created connection ) will be closed by Azure Service Bus
So, for a long-running application which is very busy at daytime ( 1000 message per second ) and very low traffic at night ( 1 message per hour ), what is the strategy here?
first and foremost, whenever that message comes, template.send(message) must success and the message is delivered to the Azure Service Bus.
so, it leaves us with two options:
- we keep the link and/or connection open and active, probably by sending some heartbeat message to Service Bus in a 10 minutes at most period
- we accept the server's choice and let it close the idle link and connection, but open a new one to deliver the message when it comes, but in this case, when the server close the link / connection, it should NOT be an error log to the end user, since by selecting this strategy, a closed connection is not an error anymore.
yiliuTo
zoladkow
nescadajci
Netyyyy
neffsvg
Saljack
vinaysurya
dmuley17
Describe the bug When using
JMSTemplateto send message to Service Bus, it throws exception if the connection is idle more than 10 minutesanother issue. is the performance, it takes 5 seconds to send the first message, see the log below for more detials
Exception or Stack Trace
To Reproduce
Add the code snippet that causes the issue.
it can be reproducecd by this code
I'm using
Expected behavior
There are two problems with this issue:
the underlying connection is not kept alive
with tracing log enabled, we can see, there is a
IdleTimeoutCheckis running every 60 seconds to try to keep the connection alive by sending an Empty Frame, but seems this doesn't keep the connection alive, so how to keep the connection alive? is it related to https://github.com/Azure/azure-sdk-for-python/pull/10209?org.springframework.jms.connection.SingleConnectionFactory#reconnectOnExceptionis not honoured.Spring Cloud Azure uses
CachingConnectionFactoryby default, according to its javadocFrom the current behavior of this exception, it doesn't auto recovery from the underlying connection.
I have the full log attached below
first message delivered! why it take 4 seconds to send a single message??? can we do something to speed up this? warmup during bootstrap process?
--- seems Server send this message every 3 minutes
---> looks like this IdleTimeoutCheck doesn't work, maybe Azuer Service bus doesn't expect an empty frame?
--> Now in LINK_FINAL state, takes 10 minutes, is it a hard limit from Service Bus? is there a doc saying that? https://github.com/Azure/azure-sdk-for-python/issues/10127 https://github.com/Azure/azure-sdk-for-python/pull/10209
--> About to send a new message, the first one was sent in 2022-11-06 10:57:01.630, and we have a 15 minutes scheduler