Impact of Robot Framework update from version 4.1.3 to version 6.0.2

[HolQue]

Observation: In case of issues in JSON configuration files tests are executed (instead of skipped like before).

Error scenario: In JSON configuration file the mandatory parameter 'WelcomeString' is missing (test case 'TSM_0050-(VARIANT_HANDLING)-[BADCASE]').

In previous version (Robot Framework v. 4.1.3 with RobotFramework_TestsuitesManagement v. 0.6.1) this is the debug log:

INFO - +- START SETUP: tm.Testsuite Setup [ ./config/tsm-test_variants.json ]
ERROR - The parameter 'WelcomeString' is a required property, but it's not set in JSON configuration file.
UNKNOWN - JSON schema validation failed!
DEBUG - JSON schema validation failed!
INFO - +- END SETUP: tm.Testsuite Setup (27)
------------------------------------------------------
INFO - +- START TEST: Test Case tsm-testfile-02 [ ]
------------------------------------------------------
INFO - +- END TEST: Test Case tsm-testfile-02 (1)
------------------------------------------------------
INFO - +- START TEARDOWN: tm.Testsuite Teardown [ ]
INFO - testsuite_teardown: Will be implemented later
INFO - +- END TEARDOWN: tm.Testsuite Teardown (1)

It can be seen that no test code is executed (like expected).

In new version (Robot Framework v. 6.0.2 with RobotFramework_TestsuitesManagement v. 0.6.1) this is the debug log:

INFO - +- START SETUP: tm.Testsuite Setup [ ./config/tsm-test_variants.json ]
ERROR - The parameter 'WelcomeString' is a required property, but it's not set in JSON configuration file.
UNKNOWN - JSON schema validation failed!
DEBUG - JSON schema validation failed!
INFO - +- END SETUP: tm.Testsuite Setup (32)
------------------------------------------------------
INFO - +- START TEST: Test Case tsm-testfile-02 [ ]
------------------------------------------------------
INFO - +-- START SETUP: tm.Testcase Setup [ ]
INFO - Test Count: 1
INFO - +-- END SETUP: tm.Testcase Setup (0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO - +-- START KEYWORD: rf.extensions.Pretty Print [ ${CONFIG.Project} | PARAMS-VERIFIKATION : ({CONFIG.Project} / tsm-testfile-02) ]
UNKNOWN - Resolving variable '${CONFIG.Project}' failed: Variable '${CONFIG}' not found.
INFO - +-- END KEYWORD: rf.extensions.Pretty Print (2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO - +-- START KEYWORD: rf.extensions.Pretty Print [ ${teststring_common} | PARAMS-VERIFIKATION : ({teststring_common} / tsm-testfile-02) ]
INFO - +-- END KEYWORD: rf.extensions.Pretty Print (0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO - +-- START KEYWORD: rf.extensions.Pretty Print [ ${teststring_variant} | PARAMS-VERIFIKATION : ({teststring_variant} / tsm-testfile-02) ]
INFO - +-- END KEYWORD: rf.extensions.Pretty Print (0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO - +-- START KEYWORD: rf.extensions.Pretty Print [ ${teststring_bench} | PARAMS-VERIFIKATION : ({teststring_bench} / tsm-testfile-02) ]
INFO - +-- END KEYWORD: rf.extensions.Pretty Print (0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO - +-- START TEARDOWN: tm.Testcase Teardown [ ]
INFO - testcase_teardown: Will be implemented later
INFO - +-- END TEARDOWN: tm.Testcase Teardown (0)

Now the test is executed. This I would not expect.

This is the test code:

rf.extensions.pretty_print    ${CONFIG.Project}        PARAMS-VERIFIKATION : ({CONFIG.Project} / tsm-testfile-02)
rf.extensions.pretty_print    ${teststring_common}     PARAMS-VERIFIKATION : ({teststring_common} / tsm-testfile-02)
rf.extensions.pretty_print    ${teststring_variant}    PARAMS-VERIFIKATION : ({teststring_variant} / tsm-testfile-02)
rf.extensions.pretty_print    ${teststring_bench}      PARAMS-VERIFIKATION : ({teststring_bench} / tsm-testfile-02)

[HolQue]

Addendum:

Error scenario: Import of not existing library (Test case: TSM_0556-(ROBOT_CODE)-[BADCASE])

Observation: Test is executed. This is not expected - and in opposite to the behavior of the previous installation where the test is skipped.

[namsonx]

Hello @milanac030988 ,

Could you look into this issue, it occurred after robotframework core was updated to new version 6.0.2?

Thank you, Son

[test-fullautomation]

Hi Cuong, please look into this. This is an urgent issue. Thank you, Thomas

[HolQue]

Solved by https://github.com/test-fullautomation/robotframework/pull/27

After integration, issue can be closed.

[test-fullautomation]

solved and already merged with #27