feat(vow): Support reliable retry-send

[erights]

closes: #XXXX refs: #XXXX

Description

Retry idempotent eventual sends with reliable durable result vows that mask upgrade trauma.

The goal here is to support something like E(bob).foo(carol) but with several differences:

• Retry: While the promise for the result of sending .foo(carol) to bob is rejected with an UpgradeDisconnection, the message is resent, until finally the result of such a send settles in some other way, i.e., it is either fulfilled, or rejected by anything else.
• Zone abstraction: Since the point is to deal with upgrade trauma, these abstractions are designed to be scoped, somehow, to the durable zone. But can still be scoped to any zone.
• Vow return: Rather than returning a promise, it returns a vow scoped to the zone in question --- typically the durable zone.
• Components must be durable: The target, e.g., bob, and the arguments, .e.g., carol, must themselves be storable in the zone in question --- typically meaning they must be durable as well.
• Masks upgrade on either end: The vow for the result itself only settles to the non-UpgradeDisconnection settlement of some try or retry of the message send. These retries will continue over upgrades of either or both the sending side and the receiving side, until it works.
• Assumes idempotence: Because the operation is retried an unpredicable number of times, you should only do such a retry-send of operations that are idempotent enough for you not to care how many "extra" times the operation may have occurred.
  • Must be explicit: Because we cannot make such idempotence assumptions in general, we cannot automagically turn unmarked eventual-sends into retry-send. There must be an explicit optin expressed in the code.
  • All query-only ops are already idempotent: An important observation that makes retry-send more immediately applicable than expected.
• Design for idempotence: APIs intended to be reliably usable across upgrade of either side should often be designed to be idempotent, so that callers can use a retry-send, to mask these upgrade traumas. This may be especially relevant to the design of orchestration APIs.

In this first PR, you can only express a retry-send of const p = E(bob).foo(carol); by saying

const v = retry(bob, 'foo', [carol]);

but the intention is that the next PR will provide a more natural E-like syntax.

Note we already have variations on E, like heapVowE that is scoped to the heap zone and have extra static methods. We can make similar variants of E that are scoped to the durable zone and have an extra retry method. Suppose this E variant is called durableVowE.

We already have static methods on E that express variants of E, such as E.sendOnly(bob).foo(carol) saying to just send the message to bob as a one-way send without any result. Similarly, I imagine that the next PR may provide something like

const v = durableVowE.retry(bob).foo(carol);

But hopefully we can invent something shorter but just as clear.

The internal API also provides a means to cancel the retry loop, so it doesn't go on forever after it is no longer relevant. Using the precedent of E taking an options bag as second argument, I imagine something like

const v = durableVowE.retry(bob, { cancelToken }).foo(carol);

once we have adapted a cancellation token proposal such as https://github.com/tc39/proposal-cancellation . So this would be farther into the future.

Security Considerations

unpredicable resends of non-idempotent operations may be accidents threatening integrity. We currently have no in-band way to mark an operation as idempotent, and so no current means to check the consistency of these assumptions.

Scaling Considerations

Perpetual retries threaten availability and scalability. Though not much if they are only triggered by upgrade. Nevertheless, it would be nice to better support cancellation.

Documentation Considerations

Will need to be explained, including the caution to avoid retry-send on non-idempotent operations, which requires explaining idempotence.

Testing Considerations

As with the current state of async-flow, this only uses the lightweight low-fidelity upgrade testing framework. See https://github.com/Agoric/agoric-sdk/issues/9303

Upgrade Considerations

The point. Retry-sent messages will be retried across upgrade at either end until they settle to something else, masking upgrade trauma. Rather than a promise, a durable retry-send returns a reliable durable vow that itself survives upgrades.

[cloudflare-pages[bot]]

Deploying agoric-sdk with    Cloudflare Pages

Latest commit:	3152f63
Status:	✅  Deploy successful!
Preview URL:	https://96503a3a.agoric-sdk.pages.dev
Branch Preview URL:	https://markm-retrier.agoric-sdk.pages.dev

View logs

[erights]

See https://github.com/Agoric/agoric-sdk/pull/9621 for an experiment of a different approach.