Request: a simpler interface for all the test results in a single place!

[Aliazzzz]

Hi!

For #11 it would be mutually beneficial to have all results in a single place.

number of succes, number of failed, number of test-suites

all test suites; per test suite, all tests; per test, all results;

each result contains; test skipped (true/false) test name (string) test duration (time duration) test result (true = succes, false = failed) expected value, actual value, message (string)

which will look something like;

testsuites[i].tests[m].testresult.name testsuites[i].tests[m].testresult.duration testsuites[i].tests[m].testresult.result testsuites[i].tests[m].testresult.expected testsuites[i].tests[m].testresult.actual testsuites[i].tests[m].testresult.message

plus anything I have forgotten which is mandatory for generating the jUnit xml file or an internal log in the console output.

[sagatowski]

I like this idea. Will investigate.

[sagatowski]

I will create a separate function block to take care of this. All information will be stored in this class/FB prior to being sent to the logger.

[Aliazzzz]

I had a brainwave and I was thinking of an FB among the following code.

FB_TestsResultProvider IMPLEMENTS ITF_TestsResults VAR testresult : T_TestResult; // DUT containing name, duration, result, expected,actual, message etc etc. tests : ARRAY OF [1.. gvl max tests] OF T_Testresults; // number of tests END_VAR VAR_OUTPUT NumberOfTests : UINT; // number of actual tests NumberOfTestsuites : UINT; // number of actual test-suites Done : BOOL; // indicates whether all test results are collected (is this really necessary) Testsuites : ARRAY[1.. gvl max testsuites ] OF Tests; // number of test-suites END_VAR

iteration through the array is thus achieved; testsuites[ NumberOfTestSuites ].tests[ NumberOfTests ].testresult.name testsuites[ NumberOfTestSuites ].tests[ NumberOfTests ].testresult.duration testsuites[ NumberOfTestSuites ].tests[ NumberOfTests ].testresult.result testsuites[ NumberOfTestSuites ].tests[ NumberOfTests ].testresult.expected testsuites[ NumberOfTestSuites ].tests[ NumberOfTests ].testresult.actual testsuites[ NumberOfTestSuites ].tests[ NumberOfTests ].testresult.message etc, etc..

Also, this information could be provided by an Interface as properties or so; ITF_TestsResults :: Property Get Done : BOOL Property Get NumberOfTests : UINT Property Get NumberOfTestsuites : UINT Property Get Testsuites : ARRAY[1.. gvl max testsuites ] OF Tests;

Offcourse, if someone has a smarter idea or can smoothen edges on this idea, be my guest and feel free ;-)

[sagatowski]

Solved in c067847. Added class "FB_TestResults", which does both the collection of the data and provides an interface to get it.

[Aliazzzz]

Hi,

This was to be expected....

I have to re-open this topic as not all info is present in the single structure. I The structure is at this moment an array, but it should be a struct with an array.

The current datastucture is not yet complete as it misses important pieces of information;

Missing but mandatory;

for all testsuites; NumberOfTestSuites, NumberOfTestCases, NumberOfSuccessfulTestCases, NumberOfFailedTestCases

^In the current code I have to recalculate these numbers a 2nd time (which is pointless) because
they are not centrally stored in the result struct

each test should also contain; If the test failed, the test assertion struct should contain its; assertion message, actual value, expected value if the assertion was a succes the struct can be kept empty as filling it is pointless

[Aliazzzz]

currently; TestResults : REFERENCE TO ARRAY[1..GVL_Param_CfUnit.MaxNumberOfTestSuites] OF ST_TestSuiteResult;

I have taken the libert and implemented the following in my own code as it misses a logical 'level' and some details, like assertresult and skipped.

TestSuitesResults : REFERENCE TO ST_TestSuitesResults;

ST_TestSuitesResults:
STRUCT
   NumberOfTestSuites : UINT;
   NumberOfTestCases : UINT;
   NumberOfSuccessfulTestCases : UINT;
   NumberOfFailedTestCases : UINT;
   NumberOfSkippedTests : UINT;
   TestSuiteResult : ARRAY[1..GVL_Param_CfUnit.MaxNumberOfTestSuites] OF ST_TestSuiteResult;
END_STRUCT
END_TYPE

TYPE ST_TestSuiteResult :
STRUCT
    Name : T_MaxString; // Full class name
    Identity : UINT(0..GVL_Param_CfUnit.MaxNumberOfTestSuites); // Should be      0..GVL_Param_TcUnit.MaxNumberOfTestSuites-1 but gives unknown compiler error
    NumberOfTests : UINT(0..GVL_Param_CfUnit.MaxNumberOfTestsForEachTestSuite);
    NumberOfSkippedTests : UINT;
    NumberOfFailedTests : UINT(0..GVL_Param_CfUnit.MaxNumberOfTestsForEachTestSuite);
    TestCaseResults : ARRAY[1..GVL_Param_CfUnit.MaxNumberOfTestsForEachTestSuite] OF ST_TestCaseResult;
END_STRUCT
END_TYPE

TYPE ST_TestCaseResult :
STRUCT
    TestName : T_MaxString;
    TestClassName : T_MaxString;
    Skipped : BOOL;
    AssertResult : ST_AssertResult;
END_STRUCT
END_TYPE

TYPE ST_AssertResult :
STRUCT
    Expected : U_ExpectedOrActual;
    Actual : U_ExpectedOrActual;
    Message : T_MaxString;
    TestInstancePath : T_MaxString;
END_STRUCT
END_TYPE

Feel free to comment!

[sagatowski]

Hi,

This was to be expected....

I have to re-open this topic as not all info is present in the single structure. I The structure is at this moment an array, but it should be a struct with an array.

The current datastucture is not yet complete as it misses important pieces of information;

Missing but mandatory;

for all testsuites; NumberOfTestSuites, NumberOfTestCases, NumberOfSuccessfulTestCases, NumberOfFailedTestCases

^In the current code I have to recalculate these numbers a 2nd time (which is pointless) because they are not centrally stored in the result struct

each test should also contain; If the test failed, the test assertion struct should contain its; assertion message, actual value, expected value if the assertion was a succes the struct can be kept empty as filling it is pointless

As mentioned in issue #11 , the following was implemented:

For a first implementation, only the required parameters are included (i.e. skipped, assertresult are all optional). The optional will have to wait after the other (higher priority) issues have been solved.

Regarding:

• NumberOfTestSuites,
• NumberOfTestCases,
• NumberOfSuccessfulTestCases,
• NumberOfFailedTestCases Are not missing, but are available in FB_TestResults, and you can access them through GetGeneralTestResults().

[sagatowski]

See #70 for continuation of this, which covers the optional (non mandatory) parts. The required parameters are in place now.

[sagatowski]

Anyone know whether the total number of tests in a test suite is INCLUDING the skipped tests or not? Or are these completely separate entities?