Reconnection problems when connection goes down

[vict2019]

I've downloaded the latest package and I don't see the Unauthorized access error anymore but it looks like a problem still happens when trying to reconnect. I'm seeing the following error message:

The connection was inactive for more than the allowed 60000 milliseconds and is closed by container...

And then the receivers don't reconnect and eventually get the message that something is busted. Here are more logs:

azure:event-hubs:error [connection-1] Error (context.connection.error) occurred on the amqp connection: c {
  azure:event-hubs:error   value:
  azure:event-hubs:error    [ Typed { type: [Object], value: 'amqp:connection:forced' },
  azure:event-hubs:error      Typed {
  azure:event-hubs:error        type: [Object],
  azure:event-hubs:error        value: 'The connection was inactive for more than the allowed 60000 milliseconds and is closed by container \'LinkTracker\'. TrackingId:b8f8f89478fc4cc3beea90f3e89ef207_G5, SystemTracker:gateway5, Timestamp:2019-01-06T10:35:43' },
  azure:event-hubs:error      Typed { type: [Object], value: null } ] } +45m
  azure:event-hubs:error [connection-1] state: { wasConnectionCloseCalled: false,
  azure:event-hubs:error   numSenders: 0,
  azure:event-hubs:error   numReceivers: 24 } +2ms
  azure:event-hubs:error [connection-1] connection.close() was not called from the sdk and there were some sender or receiver links or both. We should reconnect. +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +301ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +45m
  azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +2ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '759a93a3-ed16-46f6-b1fd-88b2f560bb2a' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/2'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +0ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +7ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +8ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +0ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-1] calling detached on receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '759a93a3-ed16-46f6-b1fd-88b2f560bb2a' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/2'. +0ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-1] calling detached on receiver '2731d83e-0d34-4e29-a3d2-1622cbe23788' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/3'. +1ms
  rhea-promise:error [connection-1] The receiver is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +6ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +0ms
  rhea-promise:error [connection-1] The session is open ? -> false +1ms
  azure:event-hubs:error [connection-1] Something is busted. State of Receiver '4d8a97bf-4886-46d9-9b5d-0c0bafaf83b7' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/0' is: { wasCloseInitiated: true,
  azure:event-hubs:error   receiverError: c { value: [ [Object], [Object], [Object] ] },
  azure:event-hubs:error   _receiver:
  azure:event-hubs:error    Receiver {
  azure:event-hubs:error      domain: null,
  azure:event-hubs:error      _events:
  azure:event-hubs:error       { message: [Function],
  azure:event-hubs:error         receiver_error: [Function],
  azure:event-hubs:error         receiver_close: [Function] },
  azure:event-hubs:error      _eventsCount: 3,
  azure:event-hubs:error      _maxListeners: undefined,
  azure:event-hubs:error      actionInitiated: 0,
  azure:event-hubs:error      type: 'receiver',
  azure:event-hubs:error      _session:
  azure:event-hubs:error       Session {
  azure:event-hubs:error         domain: null,
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 2,
  azure:event-hubs:error         _maxListeners: undefined,
  azure:event-hubs:error         actionInitiated: 0,
  azure:event-hubs:error         _connection: [Object],
  azure:event-hubs:error         _session: [Object] },
  azure:event-hubs:error      _link:
  azure:event-hubs:error       Receiver {
  azure:event-hubs:error         session: [Object],
  azure:event-hubs:error         connection: [Object],
  azure:event-hubs:error         name: 'e102e390-0e1f-478f-9295-a0125c1221d7',
  azure:event-hubs:error         options: [Object],
  azure:event-hubs:error         state: [Object],
  azure:event-hubs:error         issue_flow: false,
  azure:event-hubs:error         local: [Object],
  azure:event-hubs:error         remote: [Object],
  azure:event-hubs:error         delivery_count: 17686,
  azure:event-hubs:error         credit: 814,
  azure:event-hubs:error         observers: [Object],
  azure:event-hubs:error         drain: false,
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 7 },
  azure:event-hubs:error      linkOptions:
  azure:event-hubs:error       { name: 'e102e390-0e1f-478f-9295-a0125c1221d7',
  azure:event-hubs:error         autoaccept: true,
  azure:event-hubs:error         source: [Object],
  azure:event-hubs:error         credit_window: 1000,
  azure:event-hubs:error         onMessage: [Function],
  azure:event-hubs:error         onError: [Function],
  azure:event-hubs:error         onClose: [Function],
  azure:event-hubs:error         onSessionError: [Function],
  azure:event-hubs:error         onSessionClose: [Function] } } } +14ms
  azure:event-hubs:error [connection-1] Something is busted. State of Receiver 'b49b6c50-2464-4f30-8543-b0a252409804' with address 'victoryiothubdev/ConsumerGroups/eventpublisher/Partitions/1' is: { wasCloseInitiated: true,
  azure:event-hubs:error   receiverError: c { value: [ [Object], [Object], [Object] ] },

[KyotoFox]

I am seeing the same issue, no reconnection is attempted when the connection times out. I am connecting to an event hub using an IoT Hub connection string. It is very easily reproduced by e.g. disabling your WiFi connection when running locally, which causes the connection to time out.

Here are some logs, starting after I have turned my WiFi connection off (nothing happens until 2 to 3 minutes has passed). After the following log, nothing happens, even when the WiFi connection is re-enabled.

Running node v10.15.0 on macOS 10.14 with @azure/event-hubs@1.0.8

DEBUG=azure*,rhea*

  rhea:frames [connection-2]:0 -> empty +2m
  rhea:raw [connection-2] SENT: 8 0000000802000000 +2m
  rhea:events [connection-2] Connection got event: disconnected +3m
  rhea-promise:connection [connection-2] connection got event: 'disconnected'. Re-emitting the translated context. +3m
  rhea-promise:translate [connection-2] Translating the context for event: 'disconnected'. +3m
  azure:event-hubs:error [connection-2] Error (context.error) occurred on the amqp connection: { Error: read ETIMEDOUT
  azure:event-hubs:error     at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' } +3m
  azure:event-hubs:error [connection-2] state: { wasConnectionCloseCalled: false,
  azure:event-hubs:error   numSenders: 0,
  azure:event-hubs:error   numReceivers: 4 } +0ms
  azure:event-hubs:error [connection-2] connection.close() was not called from the sdk and there were some sender or receiver links or both. We should reconnect. +1ms
  azure:event-hubs:error [connection-2] calling detached on receiver '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/0'. +304ms
  rhea-promise:error [connection-2] The receiver is open ? -> false +3m
  azure:event-hubs:error [connection-2] calling detached on receiver '8816bda2-91f2-430b-9f04-722d630bfbdb' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/1'. +1ms
  rhea-promise:error [connection-2] The receiver is open ? -> false +0ms
  azure:event-hubs:error [connection-2] calling detached on receiver '7345a09f-bf55-4a69-9592-951607f47e63' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/2'. +1ms
  rhea-promise:error [connection-2] The receiver is open ? -> false +1ms
  azure:event-hubs:error [connection-2] calling detached on receiver 'e82a21a8-b6b8-4254-b426-736a79b8db57' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/3'. +0ms
  rhea-promise:error [connection-2] The receiver is open ? -> false +0ms
  rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +3m
  rhea-promise:error [connection-2] The session is open ? -> false +0ms
  rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +0ms
  rhea-promise:error [connection-2] The session is open ? -> false +0ms
  rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +0ms
  rhea-promise:error [connection-2] The session is open ? -> false +0ms
  rhea-promise:receiver [connection-2] receiver has been closed, now closing it's session. +0ms
  rhea-promise:error [connection-2] The session is open ? -> false +0ms
  azure:event-hubs:linkEntity [connection-2] StreamingReceiver '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/0' closed. +3m
  azure:event-hubs:linkEntity [connection-2] StreamingReceiver '8816bda2-91f2-430b-9f04-722d630bfbdb' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/1' closed. +0ms
  azure:event-hubs:linkEntity [connection-2] StreamingReceiver '7345a09f-bf55-4a69-9592-951607f47e63' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/2' closed. +0ms
  azure:event-hubs:linkEntity [connection-2] StreamingReceiver 'e82a21a8-b6b8-4254-b426-736a79b8db57' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/3' closed. +1ms
  azure:event-hubs:error [connection-2] Something is busted. State of Receiver '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/0' is: { wasCloseInitiated: true,
  azure:event-hubs:error   receiverError:
  azure:event-hubs:error    { Error: read ETIMEDOUT
  azure:event-hubs:error        at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
  azure:event-hubs:error   _receiver:
  azure:event-hubs:error    Receiver {
  azure:event-hubs:error      _events:
  azure:event-hubs:error       [Object: null prototype] {
  azure:event-hubs:error         message: [Function],
  azure:event-hubs:error         receiver_error: [Function],
  azure:event-hubs:error         receiver_close: [Function] },
  azure:event-hubs:error      _eventsCount: 3,
  azure:event-hubs:error      _maxListeners: undefined,
  azure:event-hubs:error      actionInitiated: 0,
  azure:event-hubs:error      type: 'receiver',
  azure:event-hubs:error      _session:
  azure:event-hubs:error       Session {
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 2,
  azure:event-hubs:error         _maxListeners: undefined,
  azure:event-hubs:error         actionInitiated: 0,
  azure:event-hubs:error         _connection: [Connection],
  azure:event-hubs:error         _session: [Session] },
  azure:event-hubs:error      _link:
  azure:event-hubs:error       Receiver {
  azure:event-hubs:error         session: [Session],
  azure:event-hubs:error         connection: [Connection],
  azure:event-hubs:error         name: '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e',
  azure:event-hubs:error         options: [Object],
  azure:event-hubs:error         state: [EndpointState],
  azure:event-hubs:error         issue_flow: false,
  azure:event-hubs:error         local: [Object],
  azure:event-hubs:error         remote: [Object],
  azure:event-hubs:error         delivery_count: 2,
  azure:event-hubs:error         credit: 998,
  azure:event-hubs:error         observers: [EventEmitter],
  azure:event-hubs:error         drain: false,
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 7 },
  azure:event-hubs:error      linkOptions:
  azure:event-hubs:error       { name: '152529fc-ddcb-4cc9-bf48-caa5fdcd4b5e',
  azure:event-hubs:error         autoaccept: true,
  azure:event-hubs:error         source: [Object],
  azure:event-hubs:error         credit_window: 1000,
  azure:event-hubs:error         onMessage: [Function],
  azure:event-hubs:error         onError: [Function],
  azure:event-hubs:error         onClose: [Function],
  azure:event-hubs:error         onSessionError: [Function],
  azure:event-hubs:error         onSessionClose: [Function] } } } +1ms
  azure:event-hubs:error [connection-2] Something is busted. State of Receiver '8816bda2-91f2-430b-9f04-722d630bfbdb' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/1' is: { wasCloseInitiated: true,
  azure:event-hubs:error   receiverError:
  azure:event-hubs:error    { Error: read ETIMEDOUT
  azure:event-hubs:error        at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
  azure:event-hubs:error   _receiver:
  azure:event-hubs:error    Receiver {
  azure:event-hubs:error      _events:
  azure:event-hubs:error       [Object: null prototype] {
  azure:event-hubs:error         message: [Function],
  azure:event-hubs:error         receiver_error: [Function],
  azure:event-hubs:error         receiver_close: [Function] },
  azure:event-hubs:error      _eventsCount: 3,
  azure:event-hubs:error      _maxListeners: undefined,
  azure:event-hubs:error      actionInitiated: 0,
  azure:event-hubs:error      type: 'receiver',
  azure:event-hubs:error      _session:
  azure:event-hubs:error       Session {
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 2,
  azure:event-hubs:error         _maxListeners: undefined,
  azure:event-hubs:error         actionInitiated: 0,
  azure:event-hubs:error         _connection: [Connection],
  azure:event-hubs:error         _session: [Session] },
  azure:event-hubs:error      _link:
  azure:event-hubs:error       Receiver {
  azure:event-hubs:error         session: [Session],
  azure:event-hubs:error         connection: [Connection],
  azure:event-hubs:error         name: '8816bda2-91f2-430b-9f04-722d630bfbdb',
  azure:event-hubs:error         options: [Object],
  azure:event-hubs:error         state: [EndpointState],
  azure:event-hubs:error         issue_flow: false,
  azure:event-hubs:error         local: [Object],
  azure:event-hubs:error         remote: [Object],
  azure:event-hubs:error         delivery_count: 8,
  azure:event-hubs:error         credit: 992,
  azure:event-hubs:error         observers: [EventEmitter],
  azure:event-hubs:error         drain: false,
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 7 },
  azure:event-hubs:error      linkOptions:
  azure:event-hubs:error       { name: '8816bda2-91f2-430b-9f04-722d630bfbdb',
  azure:event-hubs:error         autoaccept: true,
  azure:event-hubs:error         source: [Object],
  azure:event-hubs:error         credit_window: 1000,
  azure:event-hubs:error         onMessage: [Function],
  azure:event-hubs:error         onError: [Function],
  azure:event-hubs:error         onClose: [Function],
  azure:event-hubs:error         onSessionError: [Function],
  azure:event-hubs:error         onSessionClose: [Function] } } } +1ms
  azure:event-hubs:error [connection-2] Something is busted. State of Receiver '7345a09f-bf55-4a69-9592-951607f47e63' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/2' is: { wasCloseInitiated: true,
  azure:event-hubs:error   receiverError:
  azure:event-hubs:error    { Error: read ETIMEDOUT
  azure:event-hubs:error        at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
  azure:event-hubs:error   _receiver:
  azure:event-hubs:error    Receiver {
  azure:event-hubs:error      _events:
  azure:event-hubs:error       [Object: null prototype] {
  azure:event-hubs:error         message: [Function],
  azure:event-hubs:error         receiver_error: [Function],
  azure:event-hubs:error         receiver_close: [Function] },
  azure:event-hubs:error      _eventsCount: 3,
  azure:event-hubs:error      _maxListeners: undefined,
  azure:event-hubs:error      actionInitiated: 0,
  azure:event-hubs:error      type: 'receiver',
  azure:event-hubs:error      _session:
  azure:event-hubs:error       Session {
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 2,
  azure:event-hubs:error         _maxListeners: undefined,
  azure:event-hubs:error         actionInitiated: 0,
  azure:event-hubs:error         _connection: [Connection],
  azure:event-hubs:error         _session: [Session] },
  azure:event-hubs:error      _link:
  azure:event-hubs:error       Receiver {
  azure:event-hubs:error         session: [Session],
  azure:event-hubs:error         connection: [Connection],
  azure:event-hubs:error         name: '7345a09f-bf55-4a69-9592-951607f47e63',
  azure:event-hubs:error         options: [Object],
  azure:event-hubs:error         state: [EndpointState],
  azure:event-hubs:error         issue_flow: false,
  azure:event-hubs:error         local: [Object],
  azure:event-hubs:error         remote: [Object],
  azure:event-hubs:error         delivery_count: 0,
  azure:event-hubs:error         credit: 1000,
  azure:event-hubs:error         observers: [EventEmitter],
  azure:event-hubs:error         drain: false,
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 7 },
  azure:event-hubs:error      linkOptions:
  azure:event-hubs:error       { name: '7345a09f-bf55-4a69-9592-951607f47e63',
  azure:event-hubs:error         autoaccept: true,
  azure:event-hubs:error         source: [Object],
  azure:event-hubs:error         credit_window: 1000,
  azure:event-hubs:error         onMessage: [Function],
  azure:event-hubs:error         onError: [Function],
  azure:event-hubs:error         onClose: [Function],
  azure:event-hubs:error         onSessionError: [Function],
  azure:event-hubs:error         onSessionClose: [Function] } } } +1ms
  azure:event-hubs:error [connection-2] Something is busted. State of Receiver 'e82a21a8-b6b8-4254-b426-736a79b8db57' with address 'zapcloud/ConsumerGroups/testing-knut/Partitions/3' is: { wasCloseInitiated: true,
  azure:event-hubs:error   receiverError:
  azure:event-hubs:error    { Error: read ETIMEDOUT
  azure:event-hubs:error        at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' },
  azure:event-hubs:error   _receiver:
  azure:event-hubs:error    Receiver {
  azure:event-hubs:error      _events:
  azure:event-hubs:error       [Object: null prototype] {
  azure:event-hubs:error         message: [Function],
  azure:event-hubs:error         receiver_error: [Function],
  azure:event-hubs:error         receiver_close: [Function] },
  azure:event-hubs:error      _eventsCount: 3,
  azure:event-hubs:error      _maxListeners: undefined,
  azure:event-hubs:error      actionInitiated: 0,
  azure:event-hubs:error      type: 'receiver',
  azure:event-hubs:error      _session:
  azure:event-hubs:error       Session {
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 2,
  azure:event-hubs:error         _maxListeners: undefined,
  azure:event-hubs:error         actionInitiated: 0,
  azure:event-hubs:error         _connection: [Connection],
  azure:event-hubs:error         _session: [Session] },
  azure:event-hubs:error      _link:
  azure:event-hubs:error       Receiver {
  azure:event-hubs:error         session: [Session],
  azure:event-hubs:error         connection: [Connection],
  azure:event-hubs:error         name: 'e82a21a8-b6b8-4254-b426-736a79b8db57',
  azure:event-hubs:error         options: [Object],
  azure:event-hubs:error         state: [EndpointState],
  azure:event-hubs:error         issue_flow: false,
  azure:event-hubs:error         local: [Object],
  azure:event-hubs:error         remote: [Object],
  azure:event-hubs:error         delivery_count: 4,
  azure:event-hubs:error         credit: 996,
  azure:event-hubs:error         observers: [EventEmitter],
  azure:event-hubs:error         drain: false,
  azure:event-hubs:error         _events: [Object],
  azure:event-hubs:error         _eventsCount: 7 },
  azure:event-hubs:error      linkOptions:
  azure:event-hubs:error       { name: 'e82a21a8-b6b8-4254-b426-736a79b8db57',
  azure:event-hubs:error         autoaccept: true,
  azure:event-hubs:error         source: [Object],
  azure:event-hubs:error         credit_window: 1000,
  azure:event-hubs:error         onMessage: [Function],
  azure:event-hubs:error         onError: [Function],
  azure:event-hubs:error         onClose: [Function],
  azure:event-hubs:error         onSessionError: [Function],
  azure:event-hubs:error         onSessionClose: [Function] } } } +1ms

[KyotoFox]

Is anything being done to address this issue?

[ramya-rao-a]

Thanks for reporting this @vict2019 and @KyotoFox, we are looking into this and will get back as soon as we have something to share.

[ramya-rao-a]

@vict2019 and @KyotoFox Just giving an update here that we have found the root cause as why we are not retrying the connection and resuming the receiver

• Host not found is a non retryable error. Unfortunately Nodejs gives the same ENOTFOUND error for both host not found and the network being down scenarios. Therefore, we were treating the network being down as a non retryable error as well, and so recevier didnt get resumed.
• rhea is the library we use to create the AMQP connection and AMQP links for senders/receivers. The indicator for suggesting if the link was explicitly closed by us or due to external factors had a bug. This is what caused the logs you see for Something is busted

The second issue has been now fixed in rhea. We are in the process of fixing the first issue. We are targeting sometime next week to publish an update with the fixes.

Thanks for your patience

[ramya-rao-a]

@vict2019, @KyotoFox Thanks for your patience

We have just released a new version 2.0.0 for the Event Hubs library which includes the fix for the connection related problems being discussed in this issue.

We will be re-connecting and resume receiving of messages when the network goes down and then comes back up

Please keep in mind that there is a major version update in this release. This is because of a few breaking changes for which you may have to react to and make some changes in your code. Please see the changelog for the list of breaking changes and bug fixes.