fabiowmm
commented
5 months ago @SamCullen-Element the TH GUI provides essentially the same information you get from executing the python scripts by hand, as was the norm before introducing this feature (execution from the TH GUI). I have attached two log files from local runs (one manual and the other via TH) to make it easier to verify if someone feels like that. Being that way, I understand your request is not about TH itself but rather about the level of information provided by python test scripts in general. Is that right? CC @raju-apple @cecille logs_from_manual_execution.txt logs_from_th_gui_execution.txt
cecille
raju-apple
SamCullen-Element
When validating a python script during a Test Event or SVE, the python script does not provide supporting evidence to show that it is conforming to the test plan and does not provide evidence to support the reported verdict.
Description:
When reviewing the output from a python script test case there is a common theme that there is no log of the command that was issued by the test harness and no log of the returned value from a sample.
A script that was previously validated, but has been changed, excluding through the CCB process as this is slightly different, does not usually inherit its previously validated state under normal CSA rules. As a result, and following usual processes, these scripts become provisional again and need to be revalidated.
Example 1: python output.
A primary part of the requirements for a reviewer in a Test Event or SVE is to validate that the scripts going through testing are correct and the evidence for this is the log of that test case which is submitted for review. These logs are the primary evidence the CSA has historically used to show that test cases and test scripts are compliant and validated. The issue here is that the log does not provide evidence to show that the step has been run accurately and it does not show the values of attributes when read from a sample that would confirm the accuracy of the PASS/FAIL verdict presented by the script. This means that a reviewer would not be able to confidently agree with the test or its verdict, either in an event or a live certification or a recertification review.
To satisfy the proof of evidence we would need the script output log to show the command issued from a test harness, such as with Example 2. and the response data from the sample that is assessed to generate the PASS/FAIL verdict, such as with Example 3.
Example 2: evidence of issued command.
Example 3: evidence of sample response to support a verdict.