Reduce test flakiness due to resource contention and timing

[andrewa-stripe]

Summary

• Increase timeouts for some tests (go has timeouts as test runner first-class which can be used when we need fast)
• Use assert.Eventually (this would be better than the timeouts, usually, as it will enable the test to pass as soon as it can, without trading off latency OR flakiness)

Motivation

CI builds break a lot

Test plan

• [x] run tests

Rollout/monitoring/revert plan