The sleep with arbitrary duration causes non-determinism by introducing a race condition into the test suite. The delay is intended to provide time for an external service to finish processing a task in the background, but there is no guarantee that after this time spent sleeping that the task will indeed have completed.
This PR does not fix the issue, but instead works around it by increasing the sleep duration by ~50%, allowing more time for the background task to complete.
Dependencies on services that cannot be fully controlled by the test suite create non-determinism, whether directly or indirectly. Instead, there are a few strategies that can help deal with this:
Inject mock services that can be controlled by the test suite. Mock and test for all expected potential outcomes of attempting to communicate with that service.
Or use a publish/subscribe or a polling model to determine when an asynchronous task has completed, instead of waiting for a timeout to be reached. This can not only help to deal with non-determinism, but it can also speed up the test suite by short-circuiting before the timeout has been reached. At worst, the external service fails to respond, the timeout is reached, and there is still some non-determinism for whether the test will consistently pass.
canterberry
commented
6 years ago
Fixes #167.
The sleep with arbitrary duration causes non-determinism by introducing a race condition into the test suite. The delay is intended to provide time for an external service to finish processing a task in the background, but there is no guarantee that after this time spent sleeping that the task will indeed have completed.
This PR does not fix the issue, but instead works around it by increasing the sleep duration by ~50%, allowing more time for the background task to complete.
Dependencies on services that cannot be fully controlled by the test suite create non-determinism, whether directly or indirectly. Instead, there are a few strategies that can help deal with this: