Client.open() fails with "rhea: connection disconnected"

[edonosotti]

Description of the issue

All of a sudden, in one of our environments any attempt to invoke a Direct Method on any device in the fleet consistently failed. Searching into the logs we found out that the Client.open() method of the SDK failed after ~4 minutes with the following error:

{"message": "rhea: connection disconnected", "code": ""}

Notes:

• First of all, it's weird that the code property in the error message is empty.
• Instances of Client are being created and destroyed at each attempt, a fresh instance is created at any new attempt.
• Restarting the application did fix the problem, so it seems that the SDK was left in a bad state after a failed attempt and creating new Client instances in the running application was not enough to get rid of it.
• The metrics showed no exhaustion of resources in any Cloud component of the solution.
• The Diagnostic Tool for the IoT Hub targeted by this integration had a warning: "IoT Hub service event detected - We have detected that your IoT Hub likely has had maintenance events in the past 24 hours which could have led to temporary disconnection of devices [..] or other service interruptions". This notice showed up some time after we had started tracking failures.

References

This issue is also similar to this one reported for azure-iot-sdk-node in 2020. Note this comment from @anthonyvercolano:

The problem where you see d2c continuing (likely after some sort of error recovery) but c2d no longer works is likely a bug on our part. Hard to reproduce, but I've seen it before. Could at its root level be caused by transient network issues OR the service moving processing between nodes.

This was somewhat our situation. We could consistently receive device-to-cloud (D2C) messages, but weren't able to invoke Direct Methods on the devices. The comment above mentions cloud-to-device (C2D) messages, but both Direct Methods and C2D messages requires the application to connect to the IoT Hub first to send either of them.

Thank you for your time and I appreciate any advice you could share!

[edonosotti]

We are still getting these errors, intermittently, across all environments and different Hubs. Sometimes it lasts for days. This issue has been pending for 6 months now, any chance that we might get answers eventually?