search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.36k stars 2k forks source link

Transient failure renewing lease com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout. #10099

Closed uttamu2812 closed 4 years ago

uttamu2812 commented 4 years ago

Describe the bug I have started an event consumer to consume messages. The event consumer will run until we stop the process, since this is my requirement.It run well for 2 days, it consumes all the messages in real time. But after 2 -3 days I am getting lots of errors in console and all message are not getting consumed.Some of them are getting missed. This problem is happening continuously.But after restarting the process it works properly. but same thing happens again after few times. Is this issue is related to storage, which I am using or problem is in my code? After those error every time it is showing time out in the log as below.Some time all are getting consumed.It is not consistent after the error came. Please help.

Exception or Stack Trace 52401 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], condition[Error{condition=null, description='null', info=null}] 52419 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 52419 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl - receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[4294967296], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false] 52419 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[c3ad74_bb3f_G3_1586473645575], localSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '4294967296'}}, defaultOutcome=null, outcomes=null, capabilities=null}] 52440 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[c3ad74_bb3f_G3_1586473645575], remoteSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@1a9837cf}, defaultOutcome=null, outcomes=null, capabilities=null}] 52441 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - onOpenComplete - clientId[PartitionReceiverImpl06be8a-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], linkName[c3ad74_bb3f_G3_1586473645575], updated-link-credit[300], sentCredits[300] 52441 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-3] INFO com.microsoft.azure.eventprocessorhost.PartitionPump - host PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966: 3: EH client and receiver creation finished Processing ended. 52445 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-3] INFO com.microsoft.azure.eventprocessorhost.PartitionContext - host PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966: 2: setOffsetAndSequenceNumber(8589942688//27) would move backwards, ignoring Processing Started. 52460 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventprocessorhost.PartitionContext - host PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966: 3: setOffsetAndSequenceNumber(8589941656//20) would move backwards, ignoring Processing Started. Processing ended. Processing ended. 52826 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-5] INFO com.microsoft.azure.eventprocessorhost.PartitionContext - host PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966: 3: setOffsetAndSequenceNumber(8589943816//22) would move backwards, ignoring Processing Started. Processing ended. Processing ended. 1493816 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 1493816 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 1493816 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 1493816 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 1493816 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [5e1e72_f31_G15_1586473645332] 1493816 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 1493817 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1493843 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1493844 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], linkName[5e1e72_f31_G15_1586473645332], tokenRenewalFailure[Connection timed out] 1493844 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], linkName[5e1e72_f31_G15_1586473645332], tokenRenewalFailure[Connection timed out] 1493844 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[5e1e72_f31_G15_1586473645332], errorCondition[proton:io], errorDescription[Connection timed out] 1493844 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] WARN com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], linkName[5e1e72_f31_G15_1586473645332], onError: com.microsoft.azure.eventhubs.EventHubException: Connection timed out 1493844 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[5e1e72_f31_G15_1586473645332], errorCondition[proton:io], errorDescription[Connection timed out] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[5e1e72_f31_G15_1586473645332], errorCondition[proton:io], errorDescription[Connection timed out] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], state[CLOSED], remoteState[ACTIVE] 1493845 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1493846 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1493846 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 1493846 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], condition[proton:io], description[Connection timed out] 1493846 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactoryb16e14], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactoryb16e14], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactoryb16e14], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactoryb16e14], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [b85ef0_e5c_G25_1586473645234] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactoryb16e14], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpla9b5f8-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], linkName[b85ef0_e5c_G25_1586473645234], tokenRenewalFailure[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpla9b5f8-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], linkName[b85ef0_e5c_G25_1586473645234], tokenRenewalFailure[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[b85ef0_e5c_G25_1586473645234], errorCondition[proton:io], errorDescription[Connection timed out] 1494228 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] WARN com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpla9b5f8-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], linkName[b85ef0_e5c_G25_1586473645234], onError: com.microsoft.azure.eventhubs.EventHubException: Connection timed out 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[b85ef0_e5c_G25_1586473645234], errorCondition[proton:io], errorDescription[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[b85ef0_e5c_G25_1586473645234], errorCondition[proton:io], errorDescription[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], state[CLOSED], remoteState[ACTIVE] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], condition[proton:io], description[Connection timed out] 1494229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-6] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory91d9a4], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory91d9a4], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory91d9a4], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory91d9a4], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [c3ad74_bb3f_G3_1586473645575] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory91d9a4], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl06be8a-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], linkName[c3ad74_bb3f_G3_1586473645575], tokenRenewalFailure[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl06be8a-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], linkName[c3ad74_bb3f_G3_1586473645575], tokenRenewalFailure[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[c3ad74_bb3f_G3_1586473645575], errorCondition[proton:io], errorDescription[Connection timed out] 1494571 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] WARN com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl06be8a-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], linkName[c3ad74_bb3f_G3_1586473645575], onError: com.microsoft.azure.eventhubs.EventHubException: Connection timed out 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[c3ad74_bb3f_G3_1586473645575], errorCondition[proton:io], errorDescription[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[c3ad74_bb3f_G3_1586473645575], errorCondition[proton:io], errorDescription[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], state[CLOSED], remoteState[ACTIVE] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], condition[proton:io], description[Connection timed out] 1494572 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-9] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1494847 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], operationTimeout[PT1M], creating a receive link 1494848 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionInit hostname[drs-uw-ehns-r.servicebus.windows.net] 1494848 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[cbs-session], condition[Error{condition=null, description='null', info=null}] 1494848 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkLocalOpen linkName[cbs:sender], localTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1494848 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[cbs:receiver], localSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1494849 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[null], errorDescription[null] 1494849 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionBound hostname[drs-uw-ehns-r.servicebus.windows.net] 1495071 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionRemoteOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], remoteContainer[c76f4995ebe046e89a33fceef3c7c4ae_G36] 1495071 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[cbs-session], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1495071 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkRemoteOpen linkName[cbs:sender], remoteTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1495071 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[cbs:receiver], remoteSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1495090 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], condition[Error{condition=null, description='null', info=null}] 1495108 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1495108 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl - receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], action[createReceiveLink], offset[4294967296], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false] 1495108 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[ac71c6_4ae_G36_1586475088264], localSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '8589942688'}}, defaultOutcome=null, outcomes=null, capabilities=null}] 1495127 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[ac71c6_4ae_G36_1586475088264], remoteSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@eb19e78}, defaultOutcome=null, outcomes=null, capabilities=null}] 1495128 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - onOpenComplete - clientId[PartitionReceiverImpl268521-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], linkName[ac71c6_4ae_G36_1586475088264], updated-link-credit[300], sentCredits[300] 1495229 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpla9b5f8-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], operationTimeout[PT1M], creating a receive link 1495231 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionInit hostname[drs-uw-ehns-r.servicebus.windows.net] 1495231 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[cbs-session], condition[Error{condition=null, description='null', info=null}] 1495231 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkLocalOpen linkName[cbs:sender], localTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1495231 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[cbs:receiver], localSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1495231 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[null], errorDescription[null] 1495232 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionBound hostname[drs-uw-ehns-r.servicebus.windows.net] 1495329 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionRemoteOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], remoteContainer[c5e070bf8ff546758767cac75fd97ffd_G24] 1495329 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[cbs-session], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1495329 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkRemoteOpen linkName[cbs:sender], remoteTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1495329 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[cbs:receiver], remoteSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1495348 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], condition[Error{condition=null, description='null', info=null}] 1495365 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1495365 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl - receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], action[createReceiveLink], offset[4294983728], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false] 1495366 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[d3145f_ffd_G24_1586475088522], localSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '8589941000'}}, defaultOutcome=null, outcomes=null, capabilities=null}] 1495573 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl06be8a-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], operationTimeout[PT1M], creating a receive link 1495573 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionInit hostname[drs-uw-ehns-r.servicebus.windows.net] 1495573 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[cbs-session], condition[Error{condition=null, description='null', info=null}] 1495573 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkLocalOpen linkName[cbs:sender], localTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1495573 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[cbs:receiver], localSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1495573 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[null], errorDescription[null] 1495574 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionBound hostname[drs-uw-ehns-r.servicebus.windows.net] 1495609 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[d3145f_ffd_G24_1586475088522], remoteSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@792c3427}, defaultOutcome=null, outcomes=null, capabilities=null}] 1495609 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - onOpenComplete - clientId[PartitionReceiverImpla9b5f8-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/1], linkName[d3145f_ffd_G24_1586475088522], updated-link-credit[300], sentCredits[300] 1495669 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionRemoteOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], remoteContainer[b59e8c9c7e9c4fc1b31b70f12cd51586_G44] 1495669 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[cbs-session], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1495669 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkRemoteOpen linkName[cbs:sender], remoteTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1495669 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[cbs:receiver], remoteSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1495687 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], condition[Error{condition=null, description='null', info=null}] 1495704 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1495704 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl - receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], action[createReceiveLink], offset[4294967296], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false] 1495704 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[88e2d4_586_G44_1586475088860], localSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '8589943816'}}, defaultOutcome=null, outcomes=null, capabilities=null}] 1495730 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[88e2d4_586_G44_1586475088860], remoteSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@15ceff63}, defaultOutcome=null, outcomes=null, capabilities=null}] 1495730 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-8] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - onOpenComplete - clientId[PartitionReceiverImpl06be8a-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], linkName[88e2d4_586_G44_1586475088860], updated-link-credit[300], sentCredits[300] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [d24ccd_392_G13_1586473615102] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[d24ccd_392_G13_1586473615102], tokenRenewalFailure[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[d24ccd_392_G13_1586473615102], tokenRenewalFailure[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[d24ccd_392_G13_1586473615102], errorCondition[proton:io], errorDescription[Connection timed out] 1594711 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[d24ccd_392_G13_1586473615102], onError: com.microsoft.azure.eventhubs.EventHubException: Connection timed out 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[d24ccd_392_G13_1586473615102], errorCondition[proton:io], errorDescription[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[d24ccd_392_G13_1586473615102], errorCondition[proton:io], errorDescription[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], state[CLOSED], remoteState[ACTIVE] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], condition[proton:io], description[Connection timed out] 1594712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 1595712 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], operationTimeout[PT1M], creating a receive link 1595713 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionInit hostname[drs-uw-ehns-r.servicebus.windows.net] 1595713 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[cbs-session], condition[Error{condition=null, description='null', info=null}] 1595713 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkLocalOpen linkName[cbs:sender], localTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1595713 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[cbs:receiver], localSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1595713 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[null], errorDescription[null] 1595713 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionBound hostname[drs-uw-ehns-r.servicebus.windows.net] 1595920 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionRemoteOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], remoteContainer[c5e070bf8ff546758767cac75fd97ffd_G24] 1595920 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[cbs-session], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1595921 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkRemoteOpen linkName[cbs:sender], remoteTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 1595921 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[cbs:receiver], remoteSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 1595939 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], condition[Error{condition=null, description='null', info=null}] 1595955 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 1595955 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl - receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], action[createReceiveLink], offset[4294983648], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false] 1595955 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-1] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[0e5e82_ffd_G24_1586475189111], localSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '8589941664'}}, defaultOutcome=null, outcomes=null, capabilities=null}] 1595975 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[0e5e82_ffd_G24_1586475189111], remoteSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@18682f2f}, defaultOutcome=null, outcomes=null, capabilities=null}] 1595975 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - onOpenComplete - clientId[PartitionReceiverImplc473dd-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[0e5e82_ffd_G24_1586475189111], updated-link-credit[300], sentCredits[300] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [0e5e82_ffd_G24_1586475189111] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[0e5e82_ffd_G24_1586475189111], tokenRenewalFailure[Connection timed out] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[0e5e82_ffd_G24_1586475189111], errorCondition[proton:io], errorDescription[Connection timed out] 2895095 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] WARN com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[0e5e82_ffd_G24_1586475189111], onError: com.microsoft.azure.eventhubs.EventHubException: Connection timed out 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[0e5e82_ffd_G24_1586475189111], errorCondition[proton:io], errorDescription[Connection timed out] 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[0e5e82_ffd_G24_1586475189111], errorCondition[proton:io], errorDescription[Connection timed out] 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 2895096 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], state[CLOSED], remoteState[ACTIVE] 2895097 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 2895097 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 2895097 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 2895097 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], condition[proton:io], description[Connection timed out] 2895097 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-10] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 2896574 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImplc473dd-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], operationTimeout[PT1M], creating a receive link 2896575 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionInit hostname[drs-uw-ehns-r.servicebus.windows.net] 2896575 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[cbs-session], condition[Error{condition=null, description='null', info=null}] 2896575 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkLocalOpen linkName[cbs:sender], localTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 2896575 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[cbs:receiver], localSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 2896575 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[null], errorDescription[null] 2896575 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-7] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionBound hostname[drs-uw-ehns-r.servicebus.windows.net] 2896740 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionRemoteOpen: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], remoteContainer[b59e8c9c7e9c4fc1b31b70f12cd51586_G44] 2896740 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[cbs-session], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 2896740 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.SendLinkHandler - onLinkRemoteOpen linkName[cbs:sender], remoteTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] 2896740 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[cbs:receiver], remoteSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] 2896758 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], condition[Error{condition=null, description='null', info=null}] 2896776 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionRemoteOpen entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], sessionIncCapacity[0], sessionOutgoingWindow[2147483647] 2896776 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl - receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], action[createReceiveLink], offset[4294983648], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false] 2896776 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-2] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkLocalOpen linkName[b3dc0a_586_G44_1586476489932], localSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '8589941664'}}, defaultOutcome=null, outcomes=null, capabilities=null}] 2896796 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler - onLinkRemoteOpen linkName[b3dc0a_586_G44_1586476489932], remoteSource[Source{address='drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@4ee16c0b}, defaultOutcome=null, outcomes=null, capabilities=null}] 2896796 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - onOpenComplete - clientId[PartitionReceiverImplc473dd-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/0], linkName[b3dc0a_586_G44_1586476489932], updated-link-credit[300], sentCredits[300] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [ac71c6_4ae_G36_1586475088264] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], linkName[ac71c6_4ae_G36_1586475088264], tokenRenewalFailure[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[ac71c6_4ae_G36_1586475088264], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] WARN com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], receiverPath[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], linkName[ac71c6_4ae_G36_1586475088264], onError: com.microsoft.azure.eventhubs.EventHubException: Connection timed out 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[ac71c6_4ae_G36_1586475088264], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[ac71c6_4ae_G36_1586475088264], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 2930529 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], state[CLOSED], remoteState[ACTIVE] 2930530 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 2930530 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], condition[Error{condition=proton:io, description='Connection timed out', info=null}] 2930530 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 2930530 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], condition[proton:io], description[Connection timed out] 2930530 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-4] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 2932007 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.MessageReceiver - clientId[PartitionReceiverImpl268521-InternalReceiver], path[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/2], operationTimeout[PT1M], creating a receive link 2932007 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-13] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionInit hostname[drs-uw-ehns-r.servicebus.windows.net] ..... ......... ...... 21516231 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-15] INFO com.microsoft.azure.eventprocessorhost.PartitionPump - host PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966: 3: Transient failure renewing lease com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout. at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:243) at com.microsoft.azure.storage.blob.CloudBlob.renewLease(CloudBlob.java:2682) at com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager.renewLeaseInternal(AzureStorageCheckpointLeaseManager.java:514) at com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager.renewLease(AzureStorageCheckpointLeaseManager.java:497) at com.microsoft.azure.eventprocessorhost.PartitionPump.leaseRenewer(PartitionPump.java:418) at com.microsoft.azure.eventprocessorhost.PartitionPump.lambda$scheduleLeaseRenewer$11(PartitionPump.java:167) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: The client could not finish the operation within specified maximum execution timeout. at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:242) ... 12 more

ANOTHER EXCEPTION part 985173 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-d3ae3a4b-502b-4742-9a7a-7d8aaaa1deb1]-1-5] WARN com.microsoft.azure.eventprocessorhost.PartitionScanner - host PhoenixApplication-d3ae3a4b-502b-4742-9a7a-7d8aaaa1deb1: Failure getting/acquiring lease, continuing java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: com/microsoft/azure/storage/RetryInfo at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:961) at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: com/microsoft/azure/storage/RetryInfo at com.microsoft.azure.storage.RetryPolicy.evaluateRetryInfo(RetryPolicy.java:160) at com.microsoft.azure.storage.RetryExponentialRetry.evaluate(RetryExponentialRetry.java:140) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:231) at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1359) at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1326) at com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager.acquireLeaseInternal(AzureStorageCheckpointLeaseManager.java:456) at com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager.acquireLease(AzureStorageCheckpointLeaseManager.java:435) at com.microsoft.azure.eventprocessorhost.PartitionScanner.lambda$null$5(PartitionScanner.java:192) at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ... 9 more Caused by: java.lang.ClassNotFoundException: com.microsoft.azure.storage.RetryInfo at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559) ... 18 more Another log is..... 20549611 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 20549612 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 20549612 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 20549612 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 20549612 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-11] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory3e2a17], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [d47b15_bf3_G39_1586492259045]

Another error part

9179600 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[cbs-session], condition[proton:io], description[Connection timed out] 9179600 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionFinal entityName[drsrefdata-uw-eh-r/ConsumerGroups/phoenix/Partitions/3], condition[proton:io], description[Connection timed out] 9179600 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-16] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionFinal: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], errorCondition[proton:io], errorDescription[Connection timed out] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportError: hostname[drs-uw-ehns-r.servicebus.windows.net:5671], error[Connection timed out] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], error[Connection timed out] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:sender] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [cbs:receiver] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing link [458e0e_69b_G45_1586480613464] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - onConnectionError: messagingFactory[MessagingFactory6f99a1], hostname[drs-uw-ehns-r.servicebus.windows.net], closing current connection 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:sender], errorCondition[proton:io], errorDescription[Connection timed out] 9179922 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-ceef03de-cb34-4e86-9908-f801b07bb966]-1-12] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - processOnClose linkName[cbs:receiver], errorCondition[proton:io], errorDescription[Connection timed out]

ANOTHER ERROR part .......
1254988 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-8] WARN com.microsoft.azure.eventprocessorhost.PartitionPump - host PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a: 0: Closing EH receiver failed. com.microsoft.azure.eventhubs.TimeoutException: Close operation on Receive Link(a5a0aa_225_G31_1586296301962) timed out at 2020-04-07T15:11:44.585-07:00[US/Pacific-New], errorContext[NS: drs-uw-ehns-q.servicebus.windows.net, PATH: drsrefdata-uw-eh-q/ConsumerGroups/phoenix/Partitions/0, REFERENCE_ID: a5a0aa_225_G31_1586296301962, PREFETCH_COUNT: 300, LINK_CREDIT: 300, PREFETCH_Q_LEN: 0] at com.microsoft.azure.eventhubs.impl.MessageReceiver$10.run(MessageReceiver.java:689) at com.microsoft.azure.eventhubs.impl.Timer$ScheduledTask.onEvent(Timer.java:49) at com.microsoft.azure.eventhubs.impl.DispatchHandler.onTimerTask(DispatchHandler.java:9) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:233) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 1254988 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-8] INFO com.microsoft.azure.eventprocessorhost.PartitionPump - host PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a: 0: Closing EH client 1254988 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-8] INFO com.microsoft.azure.eventhubs.impl.ClientEntity - close: clientId[EventHubClientImpl47df73] 1254988 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-8] INFO com.microsoft.azure.eventhubs.impl.ClientEntity - close: clientId[MessagingFactory3c5ae4] 1254989 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-9] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionLocalClose: hostname[drs-uw-ehns-q.servicebus.windows.net:5671], errorCondition[null], errorDescription[null] 1254989 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:sender], errorCondition[null], errorDescription[null] 1254989 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - closeSession for linkName[cbs:sender], errorCondition[null], errorDescription[null] 1254989 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-9] INFO com.microsoft.azure.eventhubs.impl.BaseLinkHandler - onLinkLocalClose linkName[cbs:receiver], errorCondition[null], errorDescription[null] 1254989 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-9] INFO com.microsoft.azure.eventhubs.impl.SessionHandler - onSessionLocalClose entityName[cbs-session], condition[Error{condition=null, description='null', info=null}] 1283213 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-2] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - messagingFactory[MessagingFactory93403e], hostName[drs-uw-ehns-q.servicebus.windows.net], message[stopping the reactor because thread was interrupted or the reactor has no more events to process.] 1283214 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-7] WARN com.microsoft.azure.eventprocessorhost.PartitionPump - host PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a: 2: Closing EH client failed. com.microsoft.azure.eventhubs.TimeoutException: Closing MessagingFactory timed out. at com.microsoft.azure.eventhubs.impl.MessagingFactory$2.run(MessagingFactory.java:372) at com.microsoft.azure.eventhubs.impl.Timer$ScheduledTask.onEvent(Timer.java:49) at com.microsoft.azure.eventhubs.impl.DispatchHandler.onTimerTask(DispatchHandler.java:9) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:233) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) SAMPLE: Partition 2 is closing for reason LeaseLost 1314989 [[drsrefdata-uw-eh-q|phoenix|PhoenixApplication-934bce5b-c858-417c-8381-dfae8f856f8a]-1-5] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory - messagingFactory[MessagingFactory3c5ae4], To Reproduce 1.Run below code(Event consumer which will run until we stop the process) 2.Publish messages into same event hub an monitor every day. 3.For 1 -2 day there is no error in console and messages are consumed properly in real time. 4.Some error come in console after that all messages are not getting consumed.Some of them are missed. Some time all are getting consumed.It is not consistent after the error. 5.Restart the process again same will happened from step 4 and 5.

Code Snippet public class EventProcessorTest {

public static void main(String args[]) {

      String consumerGroupName = "xx"; 
      String namespaceName =          "yy";
      String eventHubName = "drsrefdata-uw-eh-d";
        String sasKeyName = "ListenAccessKey";
     // String eventHubConnectionString = "";
      //String  storageConnectionString =         "";
      String storageContainerName = "xxx";
      String hostNamePrefix  = "xx";
      String STRGCNN_KEYVAULT_URL= "xx";
     String EVNTHB_KEYVAULT_URL= "xxt";
    // ClientSecretKeyVaultCredential is the implementation of KeyVaultCredentials
      KeyVaultClient client = new KeyVaultClient(
              new ClientSecretKeyVaultCredential("xx", "xx"));

      // KEYVAULT_URL is the location of the keyvault to use: https://yourkeyvault.vault.azure.net
      // "testSecret" is the name of the secret in the key vault
      SecretBundle storageConnectionStringSecret = client.getSecret( STRGCNN_KEYVAULT_URL, "restaurantStorageConnectionString-d" );
     SecretBundle eventHubSASKeySecret = client.getSecret( EVNTHB_KEYVAULT_URL, "restaurantEventHubSASKey-d" );

      System.err.println(storageConnectionStringSecret.value());
      System.err.println(eventHubSASKeySecret.value());

      ConnectionStringBuilder eventHubConnectionString = new ConnectionStringBuilder().setNamespaceName(namespaceName)
                .setEventHubName(eventHubName).setSasKeyName(sasKeyName).setSasKey(eventHubSASKeySecret.value());

        EventProcessorHost host = new EventProcessorHost(EventProcessorHost.createHostName(hostNamePrefix),
                eventHubName, consumerGroupName, eventHubConnectionString.toString(), storageConnectionStringSecret.value() 
                        ,
                storageContainerName);

      System.out.println("Registering host named " + host.getHostName());
      EventProcessorOptions options = new EventProcessorOptions();
      options.setExceptionNotification(new ErrorNotificationHandler());

      try {
        host.registerEventProcessor(RestaurantEventProcessor.class,
                  options).get();
    } catch (InterruptedException e) {

        System.err.println("======================================="+e.getCause());
        e.printStackTrace();
    } catch (ExecutionException e) {
    System.err.println("======================================="+e.getCause()); 
    }

    System.out.println("End of sample");
}

public  void unRegisterConsumer()   {

}

public static Throwable findRootCaused(Throwable throwable) {
    Objects.requireNonNull(throwable);
    Throwable rootCause = throwable;
    while (rootCause.getCause() != null && rootCause.getCause() != rootCause) {
        rootCause = rootCause.getCause();
    }
    return rootCause;
}

}

public class RestaurantEventProcessor implements IEventProcessor {

private static final Logger LOGGER = Logger
        .getLogger(RestaurantEventProcessor.class);
private static DateFormat format = new SimpleDateFormat("MMMM d, yyyy", Locale.ENGLISH);

@Override
public void onOpen(PartitionContext partitionContext) {
    System.out.println("SAMPLE: Partition " + partitionContext.getPartitionId() + " is opening");
    //eventPayloadProcessor = new EventPayloadProcessor();
}

@Override
public void onClose(PartitionContext partitionContext, CloseReason closeReason) {
    System.out.println("SAMPLE: Partition " + partitionContext.getPartitionId() + " is closing for reason " + closeReason.toString());

}

// onEvents is called when events are received on this partition of the Event Hub. 
@Override
public void onEvents(PartitionContext context, Iterable<EventData> events) throws Exception
{
    LOGGER.info("Processing Started." );
    System.err.println("Processing Started." );

    for (EventData data : events)
    {
        try
        {
            /*
             * System.out.println("SAMPLE (" + context.getPartitionId() + "," +
             * data.getSystemProperties().getOffset() + "," +
             * data.getSystemProperties().getSequenceNumber() + "): " + new
             * String(data.getBytes(), "UTF8"));
             */
            LOGGER.info("=========== INCOMING MESSAGE ================");
            LOGGER.info( new String(data.getBytes(), "UTF8"));
            context.checkpoint(data);
        //  System.err.println("restaurantApiDelegate obj --"+restaurantApiDelegate);

            .... save in DB

            LOGGER.info("Successfully saved  ");
        }
        catch (Exception e)
        {
            System.out.println("Processing failed for an event: " + e.toString());
            LOGGER.error("Processing failed for an event: " + e.toString());
        }
    }

    LOGGER.info("Processing ended." );
    System.err.println("Processing ended." );
}

@Override
public void onError(PartitionContext partitionContext, Throwable throwable) {
    System.err.println("======================================="+throwable.getMessage());
    LOGGER.error(throwable.getMessage());
}

@SuppressWarnings("unchecked")
private Restaurant convertMessageToObj(String message) throws JSONException, JsonParseException, JsonMappingException, IOException, ParseException {

    ........
    return res;

}

}

Expected behavior All messages will be consumed until user stop the process.

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

Additional context Are there any problem in my code while registering the host or set the check point?

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

rickle-msft commented 4 years ago

@uttamu2812 Thank you for posting your question. The first thing I'm noticing is this portion of the stack trace: 

Caused by: java.lang.NoClassDefFoundError: com/microsoft/azure/storage/RetryInfo
at com.microsoft.azure.storage.RetryPolicy.evaluateRetryInfo(RetryPolicy.java:160)
at com.microsoft.azure.storage.RetryExponentialRetry.evaluate(RetryExponentialRetry.java:140)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:231)
at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1359)
at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1326)
at

Typically we see these NoClassDefFound errors when there are dependency conflicts, so we should start there. Can you give us the output of the command mvn dependency:tree please?

joshfree commented 4 years ago

@anuchandy could you please assist with this customer-reported issue?

@uttamu2812 as @rickle-msft mentioned, we'll need to see mvn dependency:tree next. You can find more details on this at https://github.com/Azure/azure-sdk-for-java/wiki/Frequently-Asked-Questions#dependency-conflict

anuchandy commented 4 years ago

The track1 event-hub-eph:2.4.0 module has a dependency on v8 storage, specifically version v8.0.0. It seems the application is bringing a conflicting dependency of storage that is not compatible with v8.0.0, i.e. a version which does not have RetryInfo type that exists in v8.0.0.

anuchandy commented 4 years ago

@uttamu2812 - as a side note: We encourage you to evaluate and switch to the new track2 event-hub library: https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/eventhubs/azure-messaging-eventhubs.

uttamu2812 commented 4 years ago

Most of the time I get following error. 57449346 [[drsrefdata-uw-eh-r|phoenix|PhoenixApplication-032c7212-ac29-4700-b633-68ab7d20fcb7]-1-12] INFO com.microsoft.azure.eventprocessorhost.PartitionPump - host PhoenixApplication-032c7212-ac29-4700-b633-68ab7d20fcb7: 3: Transient failure renewing lease com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout. at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:243) at com.microsoft.azure.storage.blob.CloudBlob.renewLease(CloudBlob.java:2682) at com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager.renewLeaseInternal(AzureStorageCheckpointLeaseManager.java:514) at com.microsoft.azure.eventprocessorhost.AzureStorageCheckpointLeaseManager.renewLease(AzureStorageCheckpointLeaseManager.java:497) at com.microsoft.azure.eventprocessorhost.PartitionPump.leaseRenewer(PartitionPump.java:418) at com.microsoft.azure.eventprocessorhost.PartitionPump.lambda$scheduleLeaseRenewer$11(PartitionPump.java:167) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: The client could not finish the operation within specified maximum execution timeout. at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:242) ... 12 more

Why this time out issue came?

uttamu2812 commented 4 years ago

my pom for azure is as below

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-eventhubs</artifactId>
    <!-- <version>3.0.0</version> -->
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-eventhubs-eph</artifactId>
    <!-- <version>3.0.0</version> -->
    <version>2.4.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.microsoft.aad/adal4j -->
<dependency>
    <groupId>com.microsoft.aad</groupId>
    <artifactId>adal4j</artifactId>
    <version>0.0.2</version>
</dependency>

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-keyvault</artifactId>
    <version>1.0.0</version>
</dependency>
uttamu2812 commented 4 years ago

Output of mvn dependency:tree -Dverbose is below. storagev8 is showing there. Do I need to the dependency in pom.xml explicitly?

D:\MGM\SVNCODEBASE\PHOENIX\1.36.3\phoenix>D:\MGM\apache-maven-3.6.3-bin\apache-maven-3.6.3\bin\mvn dependency:tree -Dverbose [INFO] Scanning for projects... [INFO] [INFO] -----------------------< com.mgmresorts:phoenix >----------------------- [INFO] Building Phoenix Data Grid and Backing Store 1.36.3 [INFO] --------------------------------[ war ]--------------------------------- [WARNING] The POM for org.apache.qpid:proton-j:jar:0.34.0-SNAPSHOT is missing, no dependency information available [INFO] [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ phoenix --- [WARNING] Missing POM for org.apache.qpid:proton-j:jar:0.34.0-SNAPSHOT [INFO] com.mgmresorts:phoenix:war:1.36.3 [INFO] +- tibco:jms:jar:6.1.0:compile [INFO] +- org.springframework:spring-test:jar:3.2.0.RELEASE:compile [INFO] | +- org.springframework:spring-webmvc:jar:3.2.0.RELEASE:compile [INFO] | | +- (org.springframework:spring-context:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | +- org.springframework:spring-web:jar:3.2.0.RELEASE:compile [INFO] | | | +- (org.springframework:spring-context:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | | +- (org.springframework:spring-aop:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | | +- (aopalliance:aopalliance:jar:1.0:compile - omitted for duplicate) [INFO] | | | - (org.springframework:spring-beans:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | +- (org.springframework:spring-expression:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | - (org.springframework:spring-beans:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | - (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for conflict with 3.2.16.RELEASE) [INFO] +- org.springframework:spring-context:jar:3.2.0.RELEASE:compile [INFO] | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | +- org.springframework:spring-aop:jar:3.2.0.RELEASE:compile [INFO] | | +- (aopalliance:aopalliance:jar:1.0:compile - omitted for duplicate) [INFO] | | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | | - (org.springframework:spring-beans:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | +- org.springframework:spring-expression:jar:3.2.0.RELEASE:compile [INFO] | | - (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | - org.springframework:spring-beans:jar:3.2.0.RELEASE:compile [INFO] | - (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] +- org.springframework:spring-jms:jar:3.2.0.RELEASE:compile [INFO] | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | +- (org.springframework:spring-aop:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | +- (org.springframework:spring-tx:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | +- aopalliance:aopalliance:jar:1.0:compile [INFO] | +- (org.springframework:spring-context:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | - (org.springframework:spring-beans:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] +- org.springframework:spring-core:jar:3.2.16.RELEASE:compile [INFO] | - commons-logging:commons-logging:jar:1.1.3:compile [INFO] +- org.springframework:spring-context-support:jar:3.2.0.RELEASE:compile [INFO] | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for conflict with 3.2.16.RELEASE) [INFO] | +- (org.springframework:spring-context:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] | - (org.springframework:spring-beans:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] +- org.springframework:spring-tx:jar:3.2.0.RELEASE:compile [INFO] | +- (org.springframework:spring-core:jar:3.2.0.RELEASE:compile - omitted for conflict with 3.2.16.RELEASE) [INFO] | +- (aopalliance:aopalliance:jar:1.0:compile - omitted for duplicate) [INFO] | - (org.springframework:spring-beans:jar:3.2.0.RELEASE:compile - omitted for duplicate) [INFO] +- org.quartz-scheduler:quartz:jar:1.8.6:compile [INFO] | - org.slf4j:slf4j-api:jar:1.6.0:compile [INFO] +- org.reflections:reflections:jar:0.9.9-RC1:compile [INFO] | +- (com.google.guava:guava:jar:11.0.2:compile - omitted for conflict with 13.0) [INFO] | +- org.javassist:javassist:jar:3.16.1-GA:compile [INFO] | - dom4j:dom4j:jar:1.6.1:compile [INFO] | - xml-apis:xml-apis:jar:1.0.b2:compile [INFO] +- javax.jms:jms:jar:1.1:compile [INFO] +- log4j:log4j:jar:1.2.16:compile [INFO] +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] +- commons-lang:commons-lang:jar:2.6:compile [INFO] +- org.hibernate:hibernate-core:jar:4.1.1.Final:compile [INFO] | +- antlr:antlr:jar:2.7.7:compile [INFO] | +- org.jboss.logging:jboss-logging:jar:3.1.0.GA:compile [INFO] | +- org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec:jar:1.0.0.Final:compile [INFO] | +- (dom4j:dom4j:jar:1.6.1:compile - omitted for duplicate) [INFO] | +- org.hibernate.javax.persistence:hibernate-jpa-2.0-api:jar:1.0.1.Final:compile [INFO] | +- (org.javassist:javassist:jar:3.15.0-GA:compile - omitted for conflict with 3.16.1-GA) [INFO] | - org.hibernate.common:hibernate-commons-annotations:jar:4.0.1.Final:compile [INFO] | - (org.jboss.logging:jboss-logging:jar:3.1.0.CR2:compile - omitted for conflict with 3.1.0.GA) [INFO] +- org.hibernate:hibernate-entitymanager:jar:4.1.1.Final:compile [INFO] | +- (org.jboss.logging:jboss-logging:jar:3.1.0.GA:compile - omitted for duplicate) [INFO] | +- (org.hibernate:hibernate-core:jar:4.1.1.Final:compile - omitted for duplicate) [INFO] | +- (org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec:jar:1.0.0.Final:compile - omitted for duplicate) [INFO] | +- (dom4j:dom4j:jar:1.6.1:compile - omitted for duplicate) [INFO] | +- (org.hibernate.javax.persistence:hibernate-jpa-2.0-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | +- (org.javassist:javassist:jar:3.15.0-GA:compile - omitted for conflict with 3.16.1-GA) [INFO] | - (org.hibernate.common:hibernate-commons-annotations:jar:4.0.1.Final:compile - omitted for duplicate) [INFO] +- org.hibernate:hibernate-validator:jar:4.2.0.Final:compile [INFO] | +- javax.validation:validation-api:jar:1.0.0.GA:compile [INFO] | - (org.slf4j:slf4j-api:jar:1.6.1:compile - omitted for conflict with 1.6.0) [INFO] +- org.hibernate:hibernate-envers:jar:4.1.1.Final:compile [INFO] | +- (org.jboss.logging:jboss-logging:jar:3.1.0.GA:compile - omitted for duplicate) [INFO] | +- (org.hibernate:hibernate-core:jar:4.1.1.Final:compile - omitted for duplicate) [INFO] | +- (org.hibernate:hibernate-entitymanager:jar:4.1.1.Final:compile - omitted for duplicate) [INFO] | +- (dom4j:dom4j:jar:1.6.1:compile - omitted for duplicate) [INFO] | - (org.hibernate.common:hibernate-commons-annotations:jar:4.0.1.Final:compile - omitted for duplicate) [INFO] +- org.hibernate:hibernate-c3p0:jar:4.0.1.Final:compile [INFO] | +- c3p0:c3p0:jar:0.9.1:compile [INFO] | +- (org.jboss.logging:jboss-logging:jar:3.1.0.CR2:compile - omitted for conflict with 3.1.0.GA) [INFO] | - (org.hibernate:hibernate-core:jar:4.0.1.Final:compile - omitted for conflict with 4.1.1.Final) [INFO] +- org.hibernate:hibernate-ehcache:jar:4.1.6.Final:compile [INFO] | +- (org.jboss.logging:jboss-logging:jar:3.1.0.GA:compile - omitted for duplicate) [INFO] | +- (org.hibernate:hibernate-core:jar:4.1.6.Final:compile - omitted for conflict with 4.1.1.Final) [INFO] | - net.sf.ehcache:ehcache-core:jar:2.4.3:compile [INFO] | - (org.slf4j:slf4j-api:jar:1.6.1:compile - omitted for conflict with 1.6.0) [INFO] +- org.jmockit:jmockit:jar:1.8:compile [INFO] +- junit:junit:jar:4.11:test [INFO] | - org.hamcrest:hamcrest-core:jar:1.3:test [INFO] +- com.sun.jersey:jersey-server:jar:1.16:compile [INFO] | +- asm:asm:jar:3.1:compile [INFO] | - com.sun.jersey:jersey-core:jar:1.16:compile [INFO] +- com.sun.jersey:jersey-servlet:jar:1.16:compile [INFO] | - (com.sun.jersey:jersey-server:jar:1.16:compile - omitted for duplicate) [INFO] +- com.sun.jersey:jersey-json:jar:1.16:compile [INFO] | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | - javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:compile [INFO] | | - javax.activation:activation:jar:1.1:compile [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.2:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.2:compile [INFO] | | - (org.codehaus.jackson:jackson-core-asl:jar:1.9.2:compile - omitted for duplicate) [INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.2:compile [INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.2:compile - omitted for duplicate) [INFO] | | - (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.2:compile - omitted for duplicate) [INFO] | +- org.codehaus.jackson:jackson-xc:jar:1.9.2:compile [INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.2:compile - omitted for duplicate) [INFO] | | - (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.2:compile - omitted for duplicate) [INFO] | - (com.sun.jersey:jersey-core:jar:1.16:compile - omitted for duplicate) [INFO] +- javax.servlet:javax.servlet-api:jar:3.0.1:compile [INFO] +- commons-codec:commons-codec:jar:1.4:compile [INFO] +- commons-beanutils:commons-beanutils:jar:1.8.3:compile [INFO] | - (commons-logging:commons-logging:jar:1.1.1:compile - omitted for conflict with 1.1.3) [INFO] +- com.webcohesion.enunciate:enunciate-core-annotations:jar:2.9.1:compile [INFO] +- com.webcohesion.enunciate:enunciate-rt-util:jar:2.9.1:compile [INFO] +- org.jboss.weld.se:weld-se:jar:1.1.6.Final:compile [INFO] +- org.jboss.arquillian.junit:arquillian-junit-container:jar:1.0.1.Final:compile [INFO] | +- org.jboss.arquillian.junit:arquillian-junit-core:jar:1.0.1.Final:compile [INFO] | | - (org.jboss.arquillian.test:arquillian-test-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | +- org.jboss.arquillian.test:arquillian-test-api:jar:1.0.1.Final:compile [INFO] | | - org.jboss.arquillian.core:arquillian-core-api:jar:1.0.1.Final:compile [INFO] | +- org.jboss.arquillian.test:arquillian-test-spi:jar:1.0.1.Final:compile [INFO] | | +- org.jboss.arquillian.core:arquillian-core-spi:jar:1.0.1.Final:compile [INFO] | | | - (org.jboss.arquillian.core:arquillian-core-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | - (org.jboss.arquillian.test:arquillian-test-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | +- org.jboss.arquillian.container:arquillian-container-test-api:jar:1.0.1.Final:compile [INFO] | +- org.jboss.arquillian.container:arquillian-container-test-spi:jar:1.0.1.Final:compile [INFO] | | +- org.jboss.arquillian.container:arquillian-container-spi:jar:1.0.1.Final:compile [INFO] | | | +- (org.jboss.arquillian.core:arquillian-core-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | | +- (org.jboss.arquillian.config:arquillian-config-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | | +- (org.jboss.arquillian.config:arquillian-config-impl-base:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | | +- (org.jboss.shrinkwrap:shrinkwrap-api:jar:1.0.1:compile - omitted for duplicate) [INFO] | | | - org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-api-base:jar:2.0.0-alpha-3:compile [INFO] | | - (org.jboss.arquillian.test:arquillian-test-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | +- org.jboss.arquillian.core:arquillian-core-impl-base:jar:1.0.1.Final:compile [INFO] | | +- (org.jboss.arquillian.core:arquillian-core-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | - (org.jboss.arquillian.core:arquillian-core-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | +- org.jboss.arquillian.test:arquillian-test-impl-base:jar:1.0.1.Final:compile [INFO] | | +- (org.jboss.arquillian.test:arquillian-test-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | - (org.jboss.arquillian.test:arquillian-test-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | +- org.jboss.arquillian.container:arquillian-container-impl-base:jar:1.0.1.Final:compile [INFO] | | +- (org.jboss.arquillian.core:arquillian-core-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | +- (org.jboss.arquillian.core:arquillian-core-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | +- org.jboss.arquillian.config:arquillian-config-api:jar:1.0.1.Final:compile [INFO] | | | - (org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-api-base:jar:2.0.0-alpha-3:compile - omitted for duplicate) [INFO] | | +- org.jboss.arquillian.config:arquillian-config-impl-base:jar:1.0.1.Final:compile [INFO] | | | +- (org.jboss.arquillian.config:arquillian-config-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | | +- (org.jboss.arquillian.core:arquillian-core-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | | +- (org.jboss.arquillian.core:arquillian-core-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | | - (org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-spi:jar:2.0.0-alpha-3:compile - omitted for duplicate) [INFO] | | +- (org.jboss.arquillian.container:arquillian-container-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | - org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-spi:jar:2.0.0-alpha-3:compile [INFO] | | - (org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-api-base:jar:2.0.0-alpha-3:compile - omitted for duplicate) [INFO] | +- org.jboss.arquillian.container:arquillian-container-test-impl-base:jar:1.0.1.Final:compile [INFO] | | +- (org.jboss.arquillian.container:arquillian-container-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | +- (org.jboss.arquillian.test:arquillian-test-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | +- (org.jboss.arquillian.container:arquillian-container-test-api:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | | - (org.jboss.arquillian.container:arquillian-container-test-spi:jar:1.0.1.Final:compile - omitted for duplicate) [INFO] | - org.jboss.shrinkwrap:shrinkwrap-impl-base:jar:1.0.1:compile [INFO] | +- org.jboss.shrinkwrap:shrinkwrap-api:jar:1.0.1:compile [INFO] | - org.jboss.shrinkwrap:shrinkwrap-spi:jar:1.0.1:compile [INFO] | - (org.jboss.shrinkwrap:shrinkwrap-api:jar:1.0.1:compile - omitted for duplicate) [INFO] +- org.jboss.arquillian.container:arquillian-weld-embedded:jar:1.0.0.Alpha2:test [INFO] | +- org.jboss.arquillian:arquillian-spi:jar:1.0.0.Alpha2:test [INFO] | | - org.jboss.arquillian:arquillian-api:jar:1.0.0.Alpha2:test [INFO] | | - (org.jboss.shrinkwrap:shrinkwrap-api:jar:1.0.0-alpha-9:test - omitted for conflict with 1.0.1) [INFO] | +- org.jboss.arquillian.protocol:arquillian-protocol-local:jar:1.0.0.Alpha2:test [INFO] | | - (org.jboss.arquillian:arquillian-spi:jar:1.0.0.Alpha2:test - omitted for duplicate) [INFO] | +- org.jboss.arquillian.packager:arquillian-packager-applicationarchive:jar:1.0.0.Alpha2:test [INFO] | | +- (org.jboss.arquillian:arquillian-spi:jar:1.0.0.Alpha2:test - omitted for duplicate) [INFO] | | +- (org.jboss.shrinkwrap:shrinkwrap-impl-base:jar:1.0.0-alpha-9:test - omitted for conflict with 1.0.1) [INFO] | | - (org.jboss.shrinkwrap:shrinkwrap-spi:jar:1.0.0-alpha-9:test - omitted for conflict with 1.0.1) [INFO] | - org.jboss.arquillian.testenricher:arquillian-testenricher-cdi:jar:1.0.0.Alpha2:test [INFO] | - (org.jboss.arquillian:arquillian-spi:jar:1.0.0.Alpha2:test - omitted for duplicate) [INFO] +- org.slf4j:slf4j-simple:jar:1.6.6:compile [INFO] | - (org.slf4j:slf4j-api:jar:1.6.6:compile - omitted for conflict with 1.6.0) [INFO] +- com.google.guava:guava:jar:13.0:compile [INFO] +- org.hsqldb:hsqldb:jar:2.2.8:compile [INFO] +- com.oracle:jdbc:jar:11.2.0.3:compile [INFO] +- com.eaio.uuid:uuid:jar:3.2:compile [INFO] +- joda-time:joda-time:jar:2.1:compile [INFO] +- com.microsoft.azure:azure-eventhubs:jar:2.2.0:compile [INFO] | +- org.apache.qpid:proton-j:jar:0.31.0:compile [INFO] | +- com.microsoft.azure:qpid-proton-j-extensions:jar:1.1.0:compile [INFO] | | - (org.apache.qpid:proton-j:jar:0.31.0:compile - omitted for duplicate) [INFO] | - (org.slf4j:slf4j-api:jar:1.7.25:compile - omitted for conflict with 1.6.0) [INFO] +- com.microsoft.azure:azure-eventhubs-eph:jar:2.4.0:compile [INFO] | +- (com.microsoft.azure:azure-eventhubs:jar:2.2.0:compile - omitted for duplicate) [INFO] | +- com.microsoft.azure:azure-storage:jar:8.0.0:compile [INFO] | | +- com.fasterxml.jackson.core:jackson-core:jar:2.9.4:compile [INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.12:compile - omitted for conflict with 1.6.0) [INFO] | | +- org.apache.commons:commons-lang3:jar:3.4:compile [INFO] | | - com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile [INFO] | | +- (org.apache.commons:commons-lang3:jar:3.4:compile - omitted for duplicate) [INFO] | | - (com.google.guava:guava:jar:20.0:compile - omitted for conflict with 13.0) [INFO] | - com.google.code.gson:gson:jar:2.8.5:compile [INFO] +- com.microsoft.aad:adal4j:jar:0.0.2:compile [INFO] | +- com.nimbusds:oauth2-oidc-sdk:jar:4.5:compile [INFO] | | +- javax.mail:mail:jar:1.4.7:compile [INFO] | | | - (javax.activation:activation:jar:1.1:compile - omitted for duplicate) [INFO] | | +- net.jcip:jcip-annotations:jar:1.0:compile [INFO] | | +- (org.apache.commons:commons-lang3:jar:3.3.1:compile - omitted for conflict with 3.4) [INFO] | | +- (commons-codec:commons-codec:jar:1.9:compile - omitted for conflict with 1.4) [INFO] | | +- net.minidev:json-smart:jar:1.1.1:compile [INFO] | | +- com.nimbusds:lang-tag:jar:1.4:compile [INFO] | | | - (net.minidev:json-smart:jar:1.1.1:compile - omitted for duplicate) [INFO] | | - com.nimbusds:nimbus-jose-jwt:jar:3.1.2:compile [INFO] | | +- (net.jcip:jcip-annotations:jar:1.0:compile - omitted for duplicate) [INFO] | | +- (net.minidev:json-smart:jar:1.1.1:compile - omitted for duplicate) [INFO] | | - org.bouncycastle:bcprov-jdk15on:jar:1.51:compile [INFO] | +- (com.google.code.gson:gson:jar:2.2.4:compile - omitted for conflict with 2.8.5) [INFO] | - (org.slf4j:slf4j-api:jar:1.7.5:compile - omitted for conflict with 1.6.0) [INFO] - com.microsoft.azure:azure-keyvault:jar:1.0.0:compile [INFO] +- com.microsoft.azure:azure-client-runtime:jar:1.0.0:compile [INFO] | - com.microsoft.rest:client-runtime:jar:1.0.0:compile [INFO] | +- (com.google.guava:guava:jar:20.0:compile - omitted for conflict with 13.0) [INFO] | +- com.squareup.retrofit2:retrofit:jar:2.1.0:compile [INFO] | | - (com.squareup.okhttp3:okhttp:jar:3.3.0:compile - omitted for conflict with 3.3.1) [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.3.1:compile [INFO] | | - com.squareup.okio:okio:jar:1.8.0:compile [INFO] | +- com.squareup.okhttp3:logging-interceptor:jar:3.3.1:compile [INFO] | | - (com.squareup.okhttp3:okhttp:jar:3.3.1:compile - omitted for duplicate) [INFO] | +- com.squareup.okhttp3:okhttp-urlconnection:jar:3.3.1:compile [INFO] | | - (com.squareup.okhttp3:okhttp:jar:3.3.1:compile - omitted for duplicate) [INFO] | +- com.squareup.retrofit2:converter-jackson:jar:2.1.0:compile [INFO] | | +- (com.squareup.retrofit2:retrofit:jar:2.1.0:compile - omitted for duplicate) [INFO] | | - (com.fasterxml.jackson.core:jackson-databind:jar:2.7.2:compile - omitted for duplicate) [INFO] | +- com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.7.2:compile [INFO] | | +- (com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile - omitted for duplicate) [INFO] | | +- (com.fasterxml.jackson.core:jackson-core:jar:2.7.2:compile - omitted for conflict with 2.9.4) [INFO] | | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.7.2:compile - omitted for duplicate) [INFO] | | - (joda-time:joda-time:jar:2.4:compile - omitted for conflict with 2.1) [INFO] | +- (org.apache.commons:commons-lang3:jar:3.4:compile - omitted for duplicate) [INFO] | +- io.reactivex:rxjava:jar:1.2.4:compile [INFO] | +- com.squareup.retrofit2:adapter-rxjava:jar:2.1.0:compile [INFO] | | +- (com.squareup.retrofit2:retrofit:jar:2.1.0:compile - omitted for duplicate) [INFO] | | - (io.reactivex:rxjava:jar:1.1.5:compile - omitted for conflict with 1.2.4) [INFO] | - (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.6.0) [INFO] - com.microsoft.azure:azure-keyvault-webkey:jar:1.0.0:compile [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.2:compile [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile [INFO] | - (com.fasterxml.jackson.core:jackson-core:jar:2.7.2:compile - omitted for conflict with 2.9.4) [INFO] +- (commons-codec:commons-codec:jar:1.10:compile - omitted for conflict with 1.4) [INFO] - (com.google.guava:guava:jar:20.0:compile - omitted for conflict with 13.0) [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15.647 s [INFO] Finished at: 2020-04-16T13:25:52+05:30 [INFO] ------------------------------------------------------------------------

D:\MGM\SVNCODEBASE\PHOENIX\1.36.3\phoenix>D:\MGM\apache-maven-3.6.3-bin\apache-maven-3.6.3\bin\mvn dependency:tree -Dverbose

anuchandy commented 4 years ago

@uttamu2812, thank you for sharing the mvn:tree, yes I see storage-sdk-v8.0.0.

Not sure why your JVM is loading a different version that causes NoClassDefFoundError. Yes, overriding v8.0.0 in your POM can help with bumping up v8.0.0 priority when resolving conflict.

But there is a caveat - are you running in any hosted environment? The environment itself may have a hard dependency on a different version of storage; in such cases, env prioritize it, and will ignore the pom override.

Regardless - In the env where you got that ClassNotDefNotFoundError, can you go back to the log and check the class-path it produces to run your application? If there is a hard override happened, then you should see a different storage version in the class-path.

Regarding your question on v8 TimeoutException in NW call: probably @rickle-msft / storage-team is the right one to comment, but I think identifying/fixing the version your env uses is needed to debug that issue.

vipinlko commented 4 years ago

I am facing the same issue and I can see storage-sdk-v8.0.0. included. Look like changing to https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/eventhubs/azure-messaging-eventhubs. will require lot of code change and testing .

Changing the dependency latest artifact . Please help with the issue

jongio commented 4 years ago

@anuchandy - Do you have an update on this? It's currently 130 days old. Thanks, Jon

anuchandy commented 4 years ago

@jongio no update from my side, above requested few things to try out but haven't heard back.

jongio commented 4 years ago

@anuchandy - Can you have a look at @vipinlko's comment and see if we should create a new issue and close this one?

anuchandy commented 4 years ago

@vipinlko, not sure you're still blocked, but can you take a look at my above comment and see anything I mentioned there apply to your environment, in short -

Are you running on a hosted env, if so, does the env (and other libs it brings-in) force a specific version of track1 storage SDK? that overrides the version your app is trying to bring-in. Does the env inject an additional path to the JVM classpath, if so, do you see any older version SDK in the classpath?

Since this issue is a couple of months old, I'm closing this one, but please feel free to open a new issue with additional details/repro steps.