Use Case Resolved: developers are wasting many hours chasing down non-deterministic Test failures
I am a Heroic developer
who wants to work on features
so that I can provide value for our users, not chasing brittle tests
Design & Implementation Notes
Find all offending tests :
Search in IntelliJ for Thread.sleep(.*)
Search in IntelliJ for all usages of retryUntilResolved
Search in IntelliJ for all usages of eu.toolchain.async.RetryPolicy
Systematically go through all the tests, identifying and fixing tests that decide to proceed in a non-deterministic way.
What is non-deterministic and what isn’t?
“Non-deterministic” probably isn’t the correct technical term, but it means a situation where a false-positive GO / Green Light / Proceed occurs.
An example. Consider com.spotify.heroic.test.AbstractSuggestBackendIT#checks. The “bad”, non-deterministic first iteration of it proceeded when any number of suggestions were returned :
…
.directTransform(result -> {
if (result.getSuggestions().isEmpty()) {
throw new IllegalStateException("No tag suggestion
…
- this meant that if only a single suggestion was returned, the test incorrectly proceeded. The deterministic fix was to pass in the expected number of suggestions. Then you know for sure you can proceed.
Timer based tests are unreliable
Use Case Resolved: developers are wasting many hours chasing down non-deterministic Test failures
Design & Implementation Notes
Thread.sleep(.*)retryUntilResolvedeu.toolchain.async.RetryPolicyWhat is non-deterministic and what isn’t?
“Non-deterministic” probably isn’t the correct technical term, but it means a situation where a false-positive
GO / Green Light / Proceedoccurs.An example. Consider
com.spotify.heroic.test.AbstractSuggestBackendIT#checks. The “bad”, non-deterministic first iteration of it proceeded when any number of suggestions were returned :- this meant that if only a single suggestion was returned, the test incorrectly proceeded. The deterministic fix was to pass in the expected number of suggestions. Then you know for sure you can proceed.