To keep the test log file "clean" we got the habit to suppress standard error of function calls which are expected to fail.
However, by now, I found myself many times uncommenting the suppression to see what that output was and, reviewing PR, I realised that some problems which were not noticed could have been if that output was not suppressed.
It might be cool and convenient to get rid of all these failures. Clearly, this would then clutter the log file so much, that a test failure would not be visible any more. One could do the following then:
Use a different logger level (like Print_Attention) to print tests failures, while waiting that the BashLogger developer (😅) adds a way to customise the logger with new levels.
Since going through all tests is already painful, this should be done only once and I'd delegate the standard pattern
if [ $? -ne 0 ]]; then
Print_Error '...'
return 1
fi
to a function, which would then use a different level. Sometimes we test for unexpected success and this new function should have an interface like Expect_Success $? 'message to be printed' 'even split' 'over lines' with possible siblings Expect_Failure with the same interface and Expect_Given_Failure which also take the expected exit code. These functions in tests should be called with a trailing || return 1 to fail the test.
Another nice improvement would be to make the runner call the Clean_Tests_Environment_For_Following_Test__* only if the test passed. At the moment we kind of do not implement these functions so much because we want files in case of failure, but then we leave them also for those tests which pass. Another alternative approach would be to make the runner remove the test dedicated folder if the test pass (of course, unless the user gave -k). This makes again the Clean* functions rarely needed, which is actually good IMO as it is less work for the developer. I'd go this way.
To keep the test log file "clean" we got the habit to suppress standard error of function calls which are expected to fail.
However, by now, I found myself many times uncommenting the suppression to see what that output was and, reviewing PR, I realised that some problems which were not noticed could have been if that output was not suppressed.
It might be cool and convenient to get rid of all these failures. Clearly, this would then clutter the log file so much, that a test failure would not be visible any more. One could do the following then:
Print_Attention) to print tests failures, while waiting that the BashLogger developer (😅) adds a way to customise the logger with new levels.to a function, which would then use a different level. Sometimes we test for unexpected success and this new function should have an interface like
Expect_Success $? 'message to be printed' 'even split' 'over lines'with possible siblingsExpect_Failurewith the same interface andExpect_Given_Failurewhich also take the expected exit code. These functions in tests should be called with a trailing|| return 1to fail the test.Another nice improvement would be to make the runner call the
Clean_Tests_Environment_For_Following_Test__*only if the test passed. At the moment we kind of do not implement these functions so much because we want files in case of failure, but then we leave them also for those tests which pass. Another alternative approach would be to make the runner remove the test dedicated folder if the test pass (of course, unless the user gave-k). This makes again theClean*functions rarely needed, which is actually good IMO as it is less work for the developer. I'd go this way.