Automated tests for command files that set #@expectedStatus other than success may not be properly indicated in the user interface, in particular when test is misconfigured.
Environment
TSTool version (e.g., 14.0.1.dev3)
Windows 10 operating system
Steps to reproduce
Run the full test suite, with over 2000 automated tests. The CheckFile/test-CheckFile-FileSizeBytes-EQ.tstool command file had #@expectedStatus Failure and a test criteria that was not met (file size is actual 60 bytes, not 55). Because the condition is not met, the code to set the status to Fail does not trigger. This test was misconfigured. No problem indicator is shown in TSTool interface, but the test result report does indicate that the test fails because the actual status (success) does not agree with the expected status (fail). The status of RunCommands command should be failure consistent with the test summary report. This would help point out that the test is misconfigured.
# Check a file for size in bytes:
# - if file size is == a value, fail
#@expectedStatus Failure
StartLog(LogFile="Results/test-CheckFile-FileSizeBytes-EQ.TSTool.log")
CheckFile(InputFile="data\file-5lines.txt",Statistic="FileSizeBytes",CheckCriteria="==",CheckValue1=55,IfCriteriaMet=Fail)
The command status view shows the following. An additional failure message needs to be added to indicate a problem.
Expected Behavior
A RunCommands command as used in a test suite that has an #@expectedStatus value that is different than the actual status should have an additional command status message generated as a failure. This will alert the user than something is wrong with the test configuration.
Actual Behavior
The test summary report is accurate. However, there is no visual indicator in the UI, which is confusing. This leads to confusion and can also lead to more test failures, even though the software is functioning accurately.
Subject of the issue
Automated tests for command files that set
#@expectedStatusother than success may not be properly indicated in the user interface, in particular when test is misconfigured.Environment
Steps to reproduce
Run the full test suite, with over 2000 automated tests. The
CheckFile/test-CheckFile-FileSizeBytes-EQ.tstoolcommand file had#@expectedStatus Failureand a test criteria that was not met (file size is actual 60 bytes, not 55). Because the condition is not met, the code to set the status to Fail does not trigger. This test was misconfigured. No problem indicator is shown in TSTool interface, but the test result report does indicate that the test fails because the actual status (success) does not agree with the expected status (fail). The status ofRunCommandscommand should be failure consistent with the test summary report. This would help point out that the test is misconfigured.The test report summary has:
The main UI is as follows (note no indicator):
The command status view shows the following. An additional failure message needs to be added to indicate a problem.
Expected Behavior
A
RunCommandscommand as used in a test suite that has an#@expectedStatusvalue that is different than the actual status should have an additional command status message generated as a failure. This will alert the user than something is wrong with the test configuration.Actual Behavior
The test summary report is accurate. However, there is no visual indicator in the UI, which is confusing. This leads to confusion and can also lead to more test failures, even though the software is functioning accurately.