"The objective of this PR is to change the behavior of how an error is detected within a test to mark it as failed.
Currently, the tests detected these errors using return codes along with set -e. The set -e makes a script (the test in this case) interrupt if any execution within the script returns an error code other than 0. This in theory sounds good since if something went wrong, the test execution was interrupted and marked as failed, but in practice, it is not so, since if a line executes several functions, only the return of the last one is counted, therefore for this common situation:
function test_a_non_existing_function_assert_empty() {
assert_empty "$(a_non_existing_function)"
}
The test no longer failed, since although a_non_existing_function returns an error 127, the assert_empty always returns 0.
With the version of this PR, instead of using the returns of the functions, the error output is used, that is, if any execution within the script writes something in stderr, it will be considered as failed.
This changes the behavior of the runner a bit since now the tests will run to the end even if something failed at the beginning, this might seem strange but right now if one of our tests has 2 assertions and the first one fails, the second one is still executed. However, as soon as something fails within the test, it is completely evaluated as failed (even if valid assertions are given afterward), so we could conclude that we have homogenized the runner's behavior.
Another change that has been made is when printing the tests that fail due to an error, previously the name of the test was printed followed by a "with error X", which went against our error reporting standard, since in the line of the test name we never printed anything else.
Now, since error codes are not used for anything, instead what is done is printing on a new line, indented and in gray (with a format very similar to the rest of the assertions) the actual error that is occurring, therefore if for example a non-existing function is called, literally in the test it will say that it has failed because a non-existing function was called."
🔖 Changes
Use stderr instead of return code to detect errors
Standardize the printing of a test that fails due to an error
Refactor acceptance tests for the new format
✅ To-do list
[x] Make sure that all the pipeline passes
[x] Make sure you have updated the documentation to reflect the changes or new features
[x] Make sure to update the CHANGELOG.md to reflect the new feature or fix
antonio-gg-dev
commented
8 months ago
📚 Description
"The objective of this PR is to change the behavior of how an error is detected within a test to mark it as failed.
Currently, the tests detected these errors using return codes along with set -e. The set -e makes a script (the test in this case) interrupt if any execution within the script returns an error code other than 0. This in theory sounds good since if something went wrong, the test execution was interrupted and marked as failed, but in practice, it is not so, since if a line executes several functions, only the return of the last one is counted, therefore for this common situation:
The test no longer failed, since although a_non_existing_function returns an error 127, the assert_empty always returns 0.
With the version of this PR, instead of using the returns of the functions, the error output is used, that is, if any execution within the script writes something in stderr, it will be considered as failed.
This changes the behavior of the runner a bit since now the tests will run to the end even if something failed at the beginning, this might seem strange but right now if one of our tests has 2 assertions and the first one fails, the second one is still executed. However, as soon as something fails within the test, it is completely evaluated as failed (even if valid assertions are given afterward), so we could conclude that we have homogenized the runner's behavior.
Another change that has been made is when printing the tests that fail due to an error, previously the name of the test was printed followed by a "with error X", which went against our error reporting standard, since in the line of the test name we never printed anything else.
Now, since error codes are not used for anything, instead what is done is printing on a new line, indented and in gray (with a format very similar to the rest of the assertions) the actual error that is occurring, therefore if for example a non-existing function is called, literally in the test it will say that it has failed because a non-existing function was called."
🔖 Changes
✅ To-do list
CHANGELOG.mdto reflect the new feature or fix