test: Retry all RPC commands to fix MacOS CI jobs

[ximinez]

High Level Overview of Change

Retry all failed RPC connections / commands in unit tests.

Context of Change

Follow up to #5120 (23991c9), which added a retry for submit commands. It improved MacOS test reliability, but other tests are failing now.

Type of Change

• [X] Bug fix (non-breaking change which fixes an issue)
• [X] Tests (you added tests for code that already exists, or your new feature included in this PR)

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (838978b) to head (34d918a).

Files with missing lines	Patch %	Lines
include/xrpl/server/detail/Door.h	0.0%	2 Missing :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/XRPLF/rippled/pull/5171?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=XRPLF) ```diff @@ Coverage Diff @@ ## develop #5171 +/- ## ========================================= - Coverage 77.9% 77.9% -0.0% ========================================= Files 782 782 Lines 66621 66619 -2 Branches 8161 8158 -3 ========================================= - Hits 51902 51873 -29 - Misses 14719 14746 +27 ``` | [Files with missing lines](https://app.codecov.io/gh/XRPLF/rippled/pull/5171?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=XRPLF) | Coverage Δ | | |---|---|---| | [include/xrpl/server/detail/Door.h](https://app.codecov.io/gh/XRPLF/rippled/pull/5171?src=pr&el=tree&filepath=include%2Fxrpl%2Fserver%2Fdetail%2FDoor.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=XRPLF#diff-aW5jbHVkZS94cnBsL3NlcnZlci9kZXRhaWwvRG9vci5o) | `79.3% <0.0%> (ø)` | | ... and [7 files with indirect coverage changes](https://app.codecov.io/gh/XRPLF/rippled/pull/5171/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=XRPLF) [](https://app.codecov.io/gh/XRPLF/rippled/pull/5171?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=XRPLF)

[ximinez]

@thejohnfreeman @vlntb I want to bring your attention to draft PR #5180. It builds on top of this PR, but adds three commits:

1. The MacOS CI jobs have 4 configurations: The combinations of release vs debug, and unity vs non-unity.
2. The Env tests cycle through a range of ports so that if the OS is slow to close a port when one test ends, the next test doesn't immediately try to open it. A general retry operation is added so that lower-level network issues can also be retried, not just failed RPC commands.
3. The MacOS CI jobs use parallel runners! (It only uses half the available CPU threads, because using all was still not reliable, but it's better than nothing!)

I would like to roll those changes into this PR so we can get the MacOS jobs fixed all in one shot. Do you think that's a good idea, or should that be a separate PR to be merged separately sometime after this one?

[thejohnfreeman]

I don't see any approvals in here yet, so I say roll them into this PR.