Enable Link Recovery

[bainian12345]

Enable link recovery for this AMQP client. Below are the major changes made to enable this feature:

• Introduced a new concept of an AmqpLinkTerminus. This object is uniquely identified by the link name and the link role under each AMQP connection.
• Introduced new APIs to allow creating a new AMQP connection with a given AmqpConnectionSettings, as well as new API to recreate an AmqpLink using a given AmqpLinkTerminus object to preserve prior link settings.
• Added checks to prevent creating AmqpLinks with the same terminus under the same connection if link recovery is enabled.
• Added mechanism to resend unsettled deliveries upon recovering a SendingAmqpLink based on the unsettled delivery map from the remote receiver peer.
• Added mechanism to skip processing aborted and already processed deliveries sent to a ReceivingAmqpLink.

To properly test the changes mentioned above, new tests and mock broker changes were added:

• Added tests to ensure that functional AmqpLinks could be created from the new link recovery API.
• Added tests to ensure the link terminus uniqueness rules were enforced under the connection scope.
• Added tests to verify the exchange of unsettled deliveries map between the sender and receiver resulted in the desired unsettled deliveries to be resent by the sender.
• Introduced a new hook to the test broker and test connection to allow tests to verify that the AMQP frames received by the connection are correct and in desired order.

Note:

• This change does not support resuming partially sent unsettled deliveries (Received delivery state). In case a delivery is under the Received delivery state, the delivery will be resent from the beginning.
• The results of exchanging unsettled deliveries between the sender and receiver follow the spec outlined by the OASIS AMQP standard section 3.4.6.

[JoshLove-msft]

Should this PR be targeting hotfix rather than main?

[JoshLove-msft]

• Added mechanism to skip processing aborted and already processed deliveries sent to a ReceivingAmqpLink.

I assume that the implication here is that non-aborted/processed deliveries will be redelivered to receivers? Is there anything we need to do from the SB SDK perspective here or will it be automatic when reconnecting to the AmqpLinkTerminus?

[JoshLove-msft]

How is message locking going to fit in here? Are there going to be service side changes to disregard lock expiration when this feature is enabled?

[bainian12345]

There will of course be a major change from the service side as well. Lock will still expire (it's not practical to hold the message in memory forever), we will still give it a hard cap from the service side. Not sure if this answer your question?

---

In reply to: 981945704

[bainian12345]

From the SDK perspective, it would still need to ensure a reconnect attempt is made to the cloud service, as this library on its own has no reconnect capabilities. This PR just handles the AMQP details on negotiating which messages should be redelivered between the service and client.

---

In reply to: 981338624

[bainian12345]

Why is that?

---

In reply to: 977570107

[JoshLove-msft]

Why is that?

In reply to: 977570107

all of the latest versions are shipped out of hotfix. Main contains a bunch of breaking changes that have not shipped.

[JoshLove-msft]

@xinchen10 are you able to review this PR?

[xinchen10]

General comments on high level design.

1. Recovery capability should be at link level. A connection may have multiple links with different roles and they may require different capabilities.
2. The link endpoint state required by link recovery should not live in connection. Even if a connection is closed, one should still be able to resume a link.
3. The library by default provides an optional volatile link endpoint state store, which can be used for resuming a link under transient errors. The user should be able to plug in their own store to handle advanced recovery scenarios. We don't have to support it right now but the design should allow for it in future.

[xinchen10]

Replaced by #230