Refactor retry logic in relayer to use retry crate

[soareschen]

Crate

relayer

Summary

This is a spin off from #712 as the scope for the original issue is too large. This issue focus only on refactoring the ad hoc retry logic that make use of MAX_RETRIES and for loop, and refactor the code to use the retry crate and the Fibonacci retry strategy.

The refactoring will take place in multiple PRs, as there are quite a few separate places that require the refactoring.

This issue does not cover the classification of errors to determine whether it is retryable or not. This is tracked by the original issue #712.

TODO

• [x] ibc_relayer::channel
  • [x] https://github.com/informalsystems/ibc-rs/blob/e6e23b0f34259b9e51f52583fc2ca718d5d369c6/relayer/src/channel.rs#L341
  • [x] https://github.com/informalsystems/ibc-rs/blob/e6e23b0f34259b9e51f52583fc2ca718d5d369c6/relayer/src/channel.rs#L368
  • [x] https://github.com/informalsystems/ibc-rs/blob/e6e23b0f34259b9e51f52583fc2ca718d5d369c6/relayer/src/channel.rs#L392
• [x] ibc_relayer::connection
  • [x] https://github.com/informalsystems/ibc-rs/blob/1f498e3cd40b6e3cc5602f4aa5ffd2560581edf3/relayer/src/connection.rs#L256
  • [x] https://github.com/informalsystems/ibc-rs/blob/1f498e3cd40b6e3cc5602f4aa5ffd2560581edf3/relayer/src/connection.rs#L273
  • [x] https://github.com/informalsystems/ibc-rs/blob/1f498e3cd40b6e3cc5602f4aa5ffd2560581edf3/relayer/src/connection.rs#L289
• [ ] ibc_relayer::link
  • [ ] https://github.com/informalsystems/ibc-rs/blob/a1e95e1443a97f5db44bf3fd3fd53a2164e587b1/relayer/src/link.rs#L398
  • [ ] https://github.com/informalsystems/ibc-rs/blob/a1e95e1443a97f5db44bf3fd3fd53a2164e587b1/relayer/src/link.rs#L596
  • [ ] https://github.com/informalsystems/ibc-rs/blob/a1e95e1443a97f5db44bf3fd3fd53a2164e587b1/relayer/src/link.rs#L815
  • [ ] https://github.com/informalsystems/ibc-rs/blob/a1e95e1443a97f5db44bf3fd3fd53a2164e587b1/relayer/src/link.rs#L857
• [ ] ibc_relayer::foreign_client
  • [ ] https://github.com/informalsystems/ibc-rs/blob/26470510f42506825a8d103452f9bbe5d98dbe99/relayer/src/foreign_client.rs#L575

Acceptance Criteria

All relayer code use retry to retry operations instead of using for loop.

---

For Admin Use

• [x] Not duplicate issue
• [x] Appropriate labels applied
• [x] Appropriate milestone (priority) applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[adizere]

I'm wondering if it's worth refactoring our custom retry logic to use retry crate, or we should we abandon this issue and live with custom for-loop retries until a more major refactor (eg async).

Couple of reasons to abandon this issue:

• @soareschen raised a point about the non-idiomatic use of Result.
• I also found it rather difficult to work with, but I haven't noticed a better option. Between a custom for-loop and retry the only advantage I see is the fancy backoff mechanism (that could be fibonacci, constant, etc.).