The fuzzy tester has a bit of added functionality not present in many testing tools: a test can be expected to pass, or expected to fail.
This is because a geocoder is a complex machine and we can't always make a certain test pass.
We often use this when creating tests for new functionality we haven't yet implemented. Marking the tests as passing then becomes a bit of a celebration of a job completed :)
On the other hand it might be because there's a known regression that we can't currently fix, and have deemed acceptable. Maybe it came along with an improvement in another area that is considered "worth it", or we just plain don't know how to fix it yet.
This leads to 4 possible outcomes for any individual test. We currently refer to them as follows:
Passed: the test was expected to pass and it did pass
Improvement: the test was not expected to pass, but it passed (usually as a result of a bug fix or new feature)
Fail: The test was expected to fail, and it failed
Regression: The test was expected to pass, but it failed. This is usually a problem and we look at these closely
The Problem
Even for those of us familiar with the fuzzy tester, remembering if it's 'regressions' or 'fails' that represent a new, unexpected failure, can sometimes be a challenge.
When viewing test output directly, the difference in color can be helpful, but depending on how the tests are run, colors in the output might not be available.
What's changed
This PR replaces 'Fails' with 'Expected Failures' so that it's very clear which of the two types of failures are which.
Background
The fuzzy tester has a bit of added functionality not present in many testing tools: a test can be expected to pass, or expected to fail.
This is because a geocoder is a complex machine and we can't always make a certain test pass.
We often use this when creating tests for new functionality we haven't yet implemented. Marking the tests as passing then becomes a bit of a celebration of a job completed :)
On the other hand it might be because there's a known regression that we can't currently fix, and have deemed acceptable. Maybe it came along with an improvement in another area that is considered "worth it", or we just plain don't know how to fix it yet.
This leads to 4 possible outcomes for any individual test. We currently refer to them as follows:
Passed: the test was expected to pass and it did pass Improvement: the test was not expected to pass, but it passed (usually as a result of a bug fix or new feature) Fail: The test was expected to fail, and it failed Regression: The test was expected to pass, but it failed. This is usually a problem and we look at these closely
The Problem
Even for those of us familiar with the fuzzy tester, remembering if it's 'regressions' or 'fails' that represent a new, unexpected failure, can sometimes be a challenge.
When viewing test output directly, the difference in color can be helpful, but depending on how the tests are run, colors in the output might not be available.
What's changed
This PR replaces 'Fails' with 'Expected Failures' so that it's very clear which of the two types of failures are which.
Before:
After: