chore: Improve test stability with constant delay for retrying

ChaoticTempest commented 3 months ago

This should improve all the stochastic tests failing due to failing triple deadline. This uses constant times for retrying and waiting on minimum amount of triples.

volovyks commented 3 months ago

Looks like the problem remains But the solution is reasonable, maybe needs tuning

ailisp commented 3 months ago

My feeling is not all related to timeoff. Sometimes tests are failed due to timeoffs, which can be improved by this PR. However sometimes certain container unexpectedly exit, could be RAM and CPU usage hit the upper bound. On my local macbook, this happens when I run multiple tests together, in docker desktop it monitors cpu and ram usage is over the limit. On the CI the resource could be more constraint so it can be problematic with even running single test at once.

The other possibility is the testing framework. I suspect the testcontainer isn't reliable in spin up/check status/communicate with containers, but I still can't reliably reproduce it.

But this PR is still helpful, and if it makes your local testing more reliable I think should merge it

ChaoticTempest commented 2 months ago

Closing this one for now until I have some time to revisit

near / mpc

chore: Improve test stability with constant delay for retrying #635