search

Azure / azure-sdk-for-js

This repository is for active development of the Azure SDK for JavaScript (NodeJS & Browser). For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/javascript/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-js.
MIT License
2.1k stars 1.21k forks source link

[Event Hub Client], ReceiverDisconnectedError, Client disconnects without any reason #25940

Closed gabrielSoudry closed 1 year ago

gabrielSoudry commented 1 year ago

Describe the bug Our pods containing our workers that consume messages restart after a certain time (randomly every 8 hours or so) and do not try to reconnect to the event hub. Similar to https://github.com/Azure/azure-sdk-for-js/issues/15893, the ownership of partitions keep changing even if every application instance is running 24*7, and the pod restart instead of trying to establish a reconnection.

azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: already has 0 events and wants to receive 1 more events
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: is open? -> true
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: setting the wait timer for 60 seconds
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: 'receiver_error' event occurred: undefined
azure:event-hubs:verbose MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: 'receiver_close' event occurred. Value for isItselfClosed on the receiver is: 'false' Value for isConnecting on the session is: 'false'
azure:core-amqp:verbose [connection-1] Error occurred for 'receiveMessage' in attempt number 1: MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_module<s/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: false,
  info: null,
  code: 'ReceiverDisconnectedError'
}
azure:event-hubs:warning An error was thrown while receiving or processing events on partition "1"
azure:event-hubs:verbose MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
[Nest] 1  - 05/22/2023, 4:41:26 PM   ERROR [Worker] {"_eventHandlers":{},"_checkpointStore":{"_containerClient":{"url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","accountName":"ststagcheckpointfrc001","pipeline":{"factories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}}],"options":{"httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[48,130,14,179,2,1,1,2,2,3,3,4,2,192,48,4,32,22,33,0,0,117,65,154,113,146,89,206,206,220,254,216,25,203,179,226,23,31,212,22,9,179,42,25,102,20,132,59,83,4,48,173,132,158,16,230,65,218,75,194,236,82,172,123,73,221,112,208,203,6,17,243,147,194,103,106,162,147,139,224,30,17,24,97,115,63,207,222,232,248,153,175,169,48,232,226,14,248,185,161,6,2,4,100,107,155,0,162,4,2,2,28,32,95,81,72,89,70,113,31,181,155,0,0,1,135,6,210,189,82,0,0,4,3,0,71,48,69,2,33,0,216,72,146,10,255,70,116,14,2,244,16,68,235,107,,122,117,114,101,5,48,134,43,112,169,206,214,238,231,78,116,143,52,104,38,231,124,152,215,165,166,203,101,94,58,139,108,16,25,20,201,199,159,107,28,164,2,4,0,173,3,2,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}}}},"storageClientContext":{"_withCredentials":false,"_httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[48,130,14,179,2,1,1,2,2,3,3,4,2,192,48,4,32,22,33,0,0,117,65,154,113,146,89,206,206,220,254,216,25,203,179,226,23,31,212,22,9,179,42,25,102,20,132,59,83,4,48,173,132,158,16,- **Package Name**:  @azure/event-hubs
1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: already has 0 events and wants to receive 1 more events
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: is open? -> true
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: setting the wait timer for 60 seconds
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: 'receiver_error' event occurred: undefined
azure:event-hubs:verbose MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: 'receiver_close' event occurred. Value for isItselfClosed on the receiver is: 'false' Value for isConnecting on the session is: 'false'
azure:core-amqp:verbose [connection-1] Error occurred for 'receiveMessage' in attempt number 1: MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_module<s/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: false,
  info: null,
  code: 'ReceiverDisconnectedError'
}
azure:event-hubs:warning An error was thrown while receiving or processing events on partition "1"
azure:event-hubs:verbose MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
[Nest] 1  - 05/22/2023, 4:41:26 PM   ERROR [Worker] {"_eventHandlers":{},"_checkpointStore":{"_containerClient":{"url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","accountName":"ststagcheckpointfrc001","pipeline":{"factories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[31]}}],"options":{"httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[48,130,14,179,2,1,1,2,2,3,3,4,2,192,48,4,32,22,33,0,0,117,65,154,113,146,89,206,206,220,254,216,25,203,179,226,23,31,212,22,9,179,42,25,102,20,132,59,83,4,48,173,132,158,16,230,65,218,75,194,236,82,172,123,73,221,112,208,203,6,17,243,147,194,103,106,162,147,139,224,30,17,24,97,115,63,207,222,232,248,153,175,169,48,232,226,14,248,185,161,6,2,4,100,107,155,0,162,4,2,2,28,32,95,81,72,8901,94,58,139,108,16,25,20,201,199,159,107,28,164,2,4,0,173,3,2,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}}}},"storageClientContext":{"_withCredentials":false,"_httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[48,130,14,179,2,1,1,2,2,3,3,4,2,192,48,4,32,22,33,0,0,117,65,154,113,146,89,206,206,220,254,216,25,203,179,226,23,31,212,22,9,01,46,110,101,116,130,28,42,46,122,50,55,46,98,108,111,98,46,115,116,111,114,97,103,101,46,97,122,117,114,101,46,110,101,116,130,28,42,46,122,50,56,46,98,108,111,98,46,115,116,111,114,97,103192,241,48,6,212,56,14,200,14,211,108,137,35,111,39,94,78,38,85,2,127,76,190,222,206,254,110,194,51,4,26,195,121,98,202,202,142,116,125,158,48,64,157,83,225,48,134,43,112,169,206,214,238,231,78,116,143,52,104,38,231,124,152,215,165,166,203,101,94,58,139,108,16,25,20,201,199,159,107,28,164,2,4,0,173,3,2,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}},"_requestPolicyOptions":{},"_requestPolicyFactories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}}],"baseUri":"{url}","url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","version":"2021-06-08"},"isHttps":true,"credential":{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}},"_containerName":"iothub","containerContext":{"client":{"_withCredentials":false,"_httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[48,130,14,179,2,1,1,2,2,3,3,4,2,192,48,4,32,22,33,0,0,117,65,154,113,146,89,206,206,220,254,216,25,203,179,226,23,31,212,22,9,179,42,25,102,20,132,59,83,4,48,173,132,158,16,230,65,218,75,194,236,82,172,123,73,221,112,208,203,6,17,243,147,194,103,106,,52,104,38,231,124,152,215,165,166,203,101,94,58,139,108,16,25,20,201,199,159,107,28,164,2,4,0,173,3,2,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}},"_requestPolicyOptions":{},"_requestPolicyFactories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}}],"baseUri":"{url}","url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","version":"2021-06-08"}}}},"_context":{"fullyQualifiedNamespace":"eventhub-prod-cntoindigo-francecentral-namespace-1.servicebus.windows.net","eventHubName":"eventhub-prod-cntoindigo-francecentral-1","consumerGroup":"testStagingProd","partitionId":"1","eventProcessorId":"460d9431-e03e-46b7-8aa2-10abd14c659e"}}
[Nest] 1  - 05/22/2023, 4:41:26 PM   ERROR [Worker] Error on partition "1": MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:1f3921f000023ed20032d7e5646b1868_G2S3_B2S3, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~16383|teststagingprod, Timestamp:2023-05-22T16:41:26
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: deleted the receiver from the client cache
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/1-917a238f-a1d8-4958-98d6-fbbad46d0bea: is closed
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/2-09621d14-873a-4539-8965-6344167d170d: 'receiver_error' event occurred: undefined
azure:event-hubs:verbose MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:ad62335600023ed1003b4ef5646b18b7_G2S3_B5S1, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~24575|teststagingprod, Timestamp:2023-05-22T16:41:35
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/2-09621d14-873a-4539-8965-6344167d170d: 'receiver_close' event occurred. Value for isItselfClosed on the receiver is: 'false' Value for isConnecting on the session is: 'false'
azure:core-amqp:verbose [connection-1] Error occurred for 'receiveMessage' in attempt number 1: MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:ad62335600023ed1003b4ef5646b18b7_G2S3_B5S1, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~24575|teststagingprod, Timestamp:2023-05-22T16:41:35
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: false,
  info: null,
  code: 'ReceiverDisconnectedError'
}
azure:event-hubs:warning An error was thrown while receiving or processing events on partition "2"
azure:event-hubs:verbose MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:ad62335600023ed1003b4ef5646b18b7_G2S3_B5S1, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~24575|teststagingprod, Timestamp:2023-05-22T16:41:35
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1086:32)
    at Receiver.onError (/usr/app/node_modules/@azure/event-hubs/dist/index.js:1129:31)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
[Nest] 1  - 05/22/2023, 4:41:35 PM   ERROR [Worker] {"_eventHandlers":{},"_checkpointStore":{"_containerClient":{"url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","accountName":"ststagcheckpointfrc001","pipeline":{"factories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}}],"options":{"httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[48,130,14,179,2,1,1,2,2,3,3,4,2,192,48,4,32,22,33,0,0,117,65,154,113,146,89,206,206,220,254,216,25,203,179,226,23,31,212,22,9,179,42,25,102,20,132,59,83,4,48,173,132,158,16,230,65,218,75,194,236,82,172,123,73,221,112,208,203,6,17,243,147,194,103,106,162,147,139,224,30,17,24,97,115,63,207,222,232,248,153,175,169,48,232,226,14,248,185,161,6,2,4,100,107,155,0,162,4,2,2,28,32,163,130,14,57202,142,116,125,158,48,64,157,83,225,48,134,43,112,169,206,214,238,231,78,116,143,52,104,38,231,124,152,215,165,166,203,101,94,58,139,108,16,25,20,201,199,159,107,28,164,2,4,0,173,3,2,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}}}},"storageClientContext":{"_withCredentials":false,"_httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[,116,130,27,42,46,122,55,46,98,108,111,98,46,115,116,111,114,97,103,101,46,97,122,117,114,101,46,110,101,116,130,27,42,46,122,56,46,98,108,111,98,46,115,116,111,114,97,103,101,46,97,122,117,114,101,46,110,101,116,130,27,42,46,122,57,46,98,108,111,98,4,46,98,108,11159,12,242,89,221,47,56,216,208,98,228,197,175,176,49,18,235,151,48,235,159,15,31,159,110,103,84,187,99,40,72,173,29,156,132,102,178,249,142,129,229,55,233,33,164,55,54,236,73,4,213,79,5,106,12,9,230,192,241,48,6,212,56,14,200,14,211,108,137,35,111,39,94,78,38,85,2,127,76,190,222,206,254,110,194,51,4,26,195,121,98,202,202,142,116,125,158,48,64,157,83,225,48,134,43,112,169,206,214,238,231,78,116,143,52,104,38,231,124,152,215,165,166,203,101,94,58,139,108,16,25,20,201,199,159,107,28,164,2,4,0,173,3,2,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}},"_requestPolicyOptions":{},"_requestPolicyFactories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}}],"baseUri":"{url}","url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","version":"2021-06-08"},"isHttps":true,"credential":{"accountName":"ststagcheckpointfrc001","accountKey":{"type":"Buffer","data":[]}},"_containerName":"iothub","containerContext":{"client":{"_withCredentials":false,"_httpClient":{"proxyAgentMap":{},"keepAliveAgents":{"httpsAgent":{"_events":{},"_eventsCount":2,"defaultPort":443,"protocol":"https:","options":{"keepAlive":true,"path":null},"requests":{},"sockets":{},"freeSockets":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":[{"_tlsOptions":{"pipe":false,"secureContext":{"context":{}},"isServer":false,"requestCert":true,"rejectUnauthorized":true},"_secureEstablished":true,"_securePending":false,"_newSessionPending":false,"_controlReleased":true,"secureConnecting":false,"_SNICallback":null,"servername":"ststagcheckpointfrc001.blob.core.windows.net","alpnProtocol":false,"authorized":true,"authorizationError":null,"encrypted":true,"_events":{"close":[null,null,null]},"_eventsCount":9,"connecting":false,"_hadError":false,"_parent":null,"_host":"ststagcheckpointfrc001.blob.core.windows.net","_closeAfterHandlingError":false,"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":true,"ended":false,"endEmitted":false,"reading":true,"constructed":true,"sync":false,"needReadable":true,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"destroyed":false,"errored":null,"closed":false,"closeEmitted":false,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":false,"ended":false,"finished":false,"destroyed":false,"decodeStrings":false,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"afterWriteTickInfo":null,"buffered":[],"bufferedIndex":0,"allBuffers":true,"allNoop":true,"pendingcb":0,"constructed":true,"prefinished":false,"errorEmitted":false,"emitClose":false,"autoDestroy":true,"errored":null,"closed":false,"closeEmitted":false},"allowHalfOpen":false,"_sockname":null,"_pendingData":null,"_pendingEncoding":"","_server":null,"ssl":{"_parent":{"reading":true,"onconnection":null},"_secureContext":{"context":{}},"reading":true},"_requestCert":true,"_rejectUnauthorized":true,"parser":null,"_httpMessage":null,"timeout":0}]},"keepAliveMsecs":1000,"keepAlive":true,"maxSockets":null,"maxFreeSockets":256,"scheduling":"lifo","maxTotalSockets":null,"totalSocketCount":1,"maxCachedSessions":100,"_sessionCache":{"map":{"ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::":{"type":"Buffer","data":[5,165,242,238,66,104,55,18,114,35,40,21,5,230,185,113,13,106,0,140,25,68,142,240,189,116,20,35,193,73,197,76,199,125,1,49,217,129,53,158,78,141,159,100,89,99,45,250,221,245,26,16,111,142,180,148,253,180,126,136,185,28,204,15,2,201,39,160,127,62,121,74,44,186,132,138,109,217,182,45,71,19,40,153,23,254,17,5,131,147,113,218,23,110,253,176,54,223,1,99,83,154,55,60,49,2,3,1,0,1,163,130,10,53,48,130,10,49,48,130,1,126,6,10,43,6,1,4,1,214,121,2,4,2,4,130,1,110,4,130,1,106,1,104,0,118,0,238,205,208,100,213,219,26,206,197,92,183,157,180,205,19,162,50,135,70,124,188,236,222,195,81,72,89,70,113,31,181,155,0,0,1,135,6,210,189,1,1]}},"list":["ststagcheckpointfrc001.blob.core.windows.net:443:::::::::::::::::::::"]}}},"cookieJar":{"version":"tough-cookie@4.0.0","storeType":"MemoryCookieStore","rejectPublicSuffixes":false,"cookies":[]}},"_requestPolicyOptions":{},"_requestPolicyFactories":[{},{},{"telemetryString":"azsdk-js-storageblob/12.10.0 (NODE-VERSION v16.18.1; Linux 5.4.0-1098-azure)"},{},{},{},{},{},{},{},{"accountName":"ststagcheckpointfrc001","accountKey":{"]}}],"baseUri":"{url}","url":"https://ststagcheckpointfrc001.blob.core.windows.net/iothub","version":"2021-06-08"}}}},"_context":{"fullyQualifiedNamespace":"eventhub-prod-cntoindigo-francecentral-namespace-1.servicebus.windows.net","eventHubName":"eventhub-prod-cntoindigo-francecentral-1","consumerGroup":"testStagingProd","partitionId":"2","eventProcessorId":"460d9431-e03e-46b7-8aa2-10abd14c659e"}}
[Nest] 1  - 05/22/2023, 4:41:35 PM   ERROR [Worker] Error on partition "2": MessagingError: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:ad62335600023ed1003b4ef5646b18b7_G2S3_B5S1, SystemTracker:eventhub-prod-cntoindigo-francecentral-namespace-1:eventhub:eventhub-prod-cntoindigo-francecentral-1~24575|teststagingprod, Timestamp:2023-05-22T16:41:35
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/2-09621d14-873a-4539-8965-6344167d170d: deleted the receiver from the client cache
azure:event-hubs:verbose [connection-1] Receiver eventhub-prod-cntoindigo-francecentral-1/ConsumerGroups/testStagingProd/Partitions/2-09621d14-873a-4539-8965-6344167d170d: is closed
azure:core-amqp:verbose [connection-1] An error occurred on the cbs sender link.. MessagingError: The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:3085ae4766244a1eb8c382adaa29e313_G2S3, SystemTracker:gateway5, Timestamp:2023-05-22T16:46:35
    at translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at Sender.<anonymous> (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1645:37)
    at Sender.emit (node:events:525:35)
    at Sender.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Sender.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Sender.emit (node:events:513:28)
    at Sender.emit (node:domain:489:12)
    at Sender.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Sender.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: true,
  info: null,
  code: 'ConnectionForcedError'
}
azure:core-amqp:verbose [connection-1] An error occurred on the cbs receiver link.. MessagingError: The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:3085ae4766244a1eb8c382adaa29e313_G2S3, SystemTracker:gateway5, Timestamp:2023-05-22T16:46:35
    at translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at Receiver.<anonymous> (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1650:37)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: true,
  info: null,
  code: 'ConnectionForcedError'
}
azure:event-hubs:verbose an error occurred on the $management sender link.. MessagingError: The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:3085ae4766244a1eb8c382adaa29e313_G2S3, SystemTracker:gateway5, Timestamp:2023-05-22T16:46:35
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at Sender.<anonymous> (/usr/app/node_modules/@azure/event-hubs/dist/index.js:2213:42)
    at Sender.emit (node:events:525:35)
    at Sender.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Sender.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Sender.emit (node:events:513:28)
    at Sender.emit (node:domain:489:12)
    at Sender.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Sender.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: true,
  info: null,
  code: 'ConnectionForcedError'
}
azure:event-hubs:verbose an error occurred on the $management receiver link.. MessagingError: The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:3085ae4766244a1eb8c382adaa29e313_G2S3, SystemTracker:gateway5, Timestamp:2023-05-22T16:46:35
    at Object.translate (/usr/app/node_modules/@azure/core-amqp/dist/index.js:1009:23)
    at Receiver.<anonymous> (/usr/app/node_modules/@azure/event-hubs/dist/index.js:2217:42)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at emit (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:132:24)
    at Object.emitEvent (/usr/app/node_modules/rhea-promise/dist/lib/util/utils.js:143:9)
    at Receiver.<anonymous> (/usr/app/node_modules/rhea-promise/dist/lib/link.js:285:25)
    at Receiver.emit (node:events:513:28)
    at Receiver.emit (node:domain:489:12)
    at Receiver.link.dispatch (/usr/app/node_modules/rhea/lib/link.js:59:37)
    at Receiver.link.on_detach (/usr/app/node_modules/rhea/lib/link.js:162:32)
    at Session.on_detach (/usr/app/node_modules/rhea/lib/session.js:747:27)
    at Connection.<computed> [as on_detach] (/usr/app/node_modules/rhea/lib/connection.js:855:30)
    at c.dispatch (/usr/app/node_modules/rhea/lib/types.js:948:33)
    at Transport.read (/usr/app/node_modules/rhea/lib/transport.js:117:36)
    at SaslClient.read (/usr/app/node_modules/rhea/lib/sasl.js:344:26)
    at Connection.input (/usr/app/node_modules/rhea/lib/connection.js:567:37)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
  retryable: true,
  info: null,
  code: 'ConnectionForcedError'
}
azure:event-hubs:verbose [connection-1] 'disconnected' event occurred on the amqp connection.
azure:event-hubs:verbose [connection-1] Accompanying error on the context.connection: c {
  value: [
    Typed { type: [TypeDesc], value: 'amqp:connection:forced' },
    Typed {
      type: [TypeDesc],
      value: "The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:3085ae4766244a1eb8c382adaa29e313_G2S3, SystemTracker:gateway5, Timestamp:2023-05-22T16:46:35"
    },
    Typed { type: [TypeDesc], value: null }
  ]
}
azure:event-hubs:verbose [connection-1] Accompanying error on the context: Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
azure:event-hubs:verbose [connection-1] Closing all open senders and receivers in the state: { wasConnectionCloseCalled: false, numSenders: 0, numReceivers: 0 }
azure:event-hubs:verbose The connection "connection-1" has been updated to "connection-2".

To Reproduce Steps to reproduce the behavior:

  1. Start 2 event hub consumer clients with checkpointstore and processing multiple partitions (e.g. 4)
  2. Keep client running and observe crash

Expected behavior

Try to reconnect and not crashing

xirzec commented 1 year ago

/cc @jeremymeng

deyaaeldeen commented 1 year ago

This error, "New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used", most commonly happens when a consumer tries to read from a partition that is already being read from from another consumer. For more details, checkout the troubleshooting guide. Generally speaking when such an error occurs, the consumer should be restarted and subscribe to a different set of partitions.

KulkarniSiddhesh commented 1 year ago

We are facing exact same issue and because of this the production drop is blocked since a month.

Has anyone aware of any work around? May be keeping some of the dependecies on previous version or so? @deyaaeldeen

KulkarniSiddhesh commented 1 year ago

NVM Got work around for now

One can resolve the issue by keeping below version configured under package.json "@azure/service-bus": "7.8.1", "@azure/core-amqp": "3.2.2", "@azure/event-hubs": "5.8.0",

image

xirzec commented 1 year ago

thank you @KulkarniSiddhesh for the extra information, we will continue to investigate.

/cc @HarshaNalluru

jeremymeng commented 1 year ago

@KulkarniSiddhesh thanks for the report! I have some questions:

  1. I assume that @azure/event-hubs 5.9.0 also have the same issue thus you downgrade to 5.8.0?
  2. does the version of @azure/service-bus matter? That is, if you use @azure/service-bus 7.9.0 and @azure/event-hubs 5.8.0, will the issue happen?
gabrielSoudry commented 1 year ago

I tried to use with no success, same behavior ownership of partitions keep changing without any reasons. "@azure/core-amqp": "3.2.2", "@azure/event-hubs": "5.8.0", We don't have "@azure/service-bus": "7.8.1",

Perhaps add more meaningful logs in debug mode of "why" my consumer lose his partitions, and add a log when the other consumer starts to get the lock of a partition he didn't have before

KulkarniSiddhesh commented 1 year ago

@jeremymeng Yes, it worked with the combination of the versions for me.

@gabrielSoudry, here is link for 7.8.1 of @azure/service-bus https://www.npmjs.com/package/@azure/service-bus/v/7.8.1

xirzec commented 1 year ago

We recently released version 5.11.1 of @azure/event-hubs to address an important bug in retrieving updated authentication tokens: https://www.npmjs.com/package/@azure/event-hubs/v/5.11.1

I'm not sure that it will totally resolve the issue in this thread, but it should at least help with narrowing the cause should it continue to reproduce. Can you give this version a try?

jeremymeng commented 1 year ago

@jeremymeng Yes, it worked with the combination of the versions for me.

@KulkarniSiddhesh Sorry it is not clear to me whether this is answer to Question 1 or Question 2, or both.

jeremymeng commented 1 year ago

@KulkarniSiddhesh could you please clarify?

deyaaeldeen commented 1 year ago

@gabrielSoudry could you please confirm if the issue has been resolved with v5.11.1?

Perhaps add more meaningful logs in debug mode of "why" my consumer lose his partitions, and add a log when the other consumer starts to get the lock of a partition he didn't have before

The why part is pretty much due to scaling or if an existing consumer dies. The SDK doesn't know why a partition becomes available, it just pings the checkpoint store to learn about the current state of the world. Please notice that a more involved orchestration can be costly. Furthermore, we already log when a consumer takes over a partition.

github-actions[bot] commented 1 year ago

Hi @gabrielSoudry. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

gabrielSoudry commented 1 year ago

Great news :) By updating Azure identity lib + move to Azure Workload Identity + updating to 5.11.1 we no longer have the problem, I'll wait another day to reconfirm and you can close the ticket.

github-actions[bot] commented 1 year ago

Hi @gabrielSoudry. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.