Crash after failed load of test configuration

[HolQue]

Test setup:

• Robot Framework core (original)
• Testsuites management adapted to original core of Robot Framework ('unknown' replaced by 'exceptoion')

Self test: VARIANT_HANDLING / BADCASE / TSM_0050 - TSM_0066

Several badcase scenarios make the load of the test configuration fail:

FAIL - Unable to load the test configuration. The test execution will be aborted!

This is like expected so far.

Issue:

The error message is followed by several call stacks. It seems that exceptions are raised, but not catched.

Below some examples:

2024-11-12 14:22:56.371165 - FAIL - Unable to load the test configuration. The test execution will be aborted!
2024-11-12 14:22:56.371165 - DEBUG - Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Config\CConfig.py", line 375, in loadCfg
    validate(instance=oJsonCfgData, schema=oJsonSchemaCfg)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\jsonschema\validators.py", line 1332, in validate
    raise error
jsonschema.exceptions.ValidationError: 'WelcomeString' is a required property

Failed validating 'required' in schema:
    {'$schema': 'http://json-schema.org/draft-07/schema#',
     '$id': './Config/configuration_schema.json',
     'title': 'robot_config',
     'description': 'ROBFW AIO configuration file in json format',
     'type': 'object',
     'properties': {'Project': {'type': 'string'},
                    'WelcomeString': {'type': 'string'},
                    'version': {'type': 'object',
                                'properties': {'majorversion': {'type': 'string'},
                                               'minorversion': {'type': 'string'},
                                               'patchversion': {'type': 'string'}}},
                    'TargetName': {'type': 'string'},
                    'Maximum_version': {'type': ['string', 'null']},
                    'Minimum_version': {'type': ['string', 'null']},
                    'params': {'type': 'object',
                               'properties': {'global': {}},
                               'required': ['global'],
                               'additionalProperties': False}},
     'required': ['Project', 'WelcomeString', 'TargetName'],
     'additionalProperties': False}

On instance:
    {'Maximum_version': '1.0.0',
     'Minimum_version': '0.6.0',
     'Project': 'tsm-test_config_missing_param',
     'TargetName': 'Device_01',
     'params': {'global': {'teststring_common': 'I am the common '
                                                'teststring valid for all '
                                                'variants and all test '
                                                'benches',
                           'teststring_bench': 'I am the teststring '
                                               'containing the default '
                                               'value for all test benches',
                           'teststring_variant': "I am the 'missing_param' "
                                                 'configuration of '
                                                 'tsm-test'}}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 75, in testsuite_setup
    TM.CTestsuitesCfg.oConfig.loadCfg(TM.CTestsuitesCfg.oConfig)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Config\CConfig.py", line 397, in loadCfg
    raise Exception
Exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\__init__.py", line 65, in run_keyword
    return DynamicCore.run_keyword(self, name, args, kwargs)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\robotlibcore.py", line 173, in run_keyword
    return self.keywords[name](*args, **(kwargs or {}))
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 85, in testsuite_setup
    raise Exception(TM.CTestsuitesCfg.oConfig.sLoadedCfgLog['unknown'])
Exception: Unable to load the test configuration. The test execution will be aborted!

[HolQue]

2024-11-12 14:22:56.688724 - FAIL - Unable to load the test configuration. The test execution will be aborted!
2024-11-12 14:22:56.702302 - DEBUG - Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1738, in jsonLoads
    self.jsonCheck = json.loads(sJsonDataUpdated,
  File "C:\workplace\RobotFramework\python39\lib\json\__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 141, in custom_scan_once
    return self._custom_scan_once(string, idx)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 105, in _custom_scan_once
    return self.parse_object((string, idx + 1), self.strict,
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 207, in JSONObject
    raise JSONDecodeError(
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 7 column 3 (char 221)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Config\CConfig.py", line 318, in loadCfg
    oJsonCfgData = oJsonPreprocessor.jsonLoad(self.sTestCfgFile)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1470, in jsonLoad
    return self.jsonLoads(sJsonData)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1750, in jsonLoads
    raise Exception(jsonException)
Exception: Expecting property name enclosed in double quotes: line 7 column 3 (char 221)
Nearby: '... ntax_error",\n  "TargetName"      : "Device_01",\n  I_AM_SYNTAX_ERROR }} ...'
In file: 'C:/workplace/ROBFW/components/robotframework-testsuitesmanagement/test/testfiles/config/tsm-test_config_syntax_error.jsonp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 75, in testsuite_setup
    TM.CTestsuitesCfg.oConfig.loadCfg(TM.CTestsuitesCfg.oConfig)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Config\CConfig.py", line 329, in loadCfg
    raise Exception
Exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\__init__.py", line 65, in run_keyword
    return DynamicCore.run_keyword(self, name, args, kwargs)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\robotlibcore.py", line 173, in run_keyword
    return self.keywords[name](*args, **(kwargs or {}))
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 85, in testsuite_setup
    raise Exception(TM.CTestsuitesCfg.oConfig.sLoadedCfgLog['unknown'])

[HolQue]

2024-11-12 14:22:57.371048 - FAIL - Unable to load the test configuration. The test execution will be aborted!
2024-11-12 14:22:57.371048 - DEBUG - Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1738, in jsonLoads
    self.jsonCheck = json.loads(sJsonDataUpdated,
  File "C:\workplace\RobotFramework\python39\lib\json\__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 141, in custom_scan_once
    return self._custom_scan_once(string, idx)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 105, in _custom_scan_once
    return self.parse_object((string, idx + 1), self.strict,
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 186, in JSONObject
    value, end = scan_once(s, end)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 105, in _custom_scan_once
    return self.parse_object((string, idx + 1), self.strict,
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 186, in JSONObject
    value, end = scan_once(s, end)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 105, in _custom_scan_once
    return self.parse_object((string, idx + 1), self.strict,
  File "C:\workplace\RobotFramework\python39\lib\json\decoder.py", line 210, in JSONObject
    result = object_pairs_hook(pairs)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 282, in __processImportFiles
    oJsonImport = self.jsonLoad(abs_path_file)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1461, in jsonLoad
    raise Exception(f"File '{jFile}' is not existing!")
Exception: File 'C:/workplace/ROBFW/components/robotframework-testsuitesmanagement/test/testfiles/config/I_AM_NOT_EXISTING.jsonp' is not existing!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Config\CConfig.py", line 318, in loadCfg
    oJsonCfgData = oJsonPreprocessor.jsonLoad(self.sTestCfgFile)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1470, in jsonLoad
    return self.jsonLoads(sJsonData)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\JsonPreprocessor\CJsonPreprocessor.py", line 1750, in jsonLoads
    raise Exception(jsonException)
Exception: File 'C:/workplace/ROBFW/components/robotframework-testsuitesmanagement/test/testfiles/config/I_AM_NOT_EXISTING.jsonp' is not existing!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 75, in testsuite_setup
    TM.CTestsuitesCfg.oConfig.loadCfg(TM.CTestsuitesCfg.oConfig)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Config\CConfig.py", line 329, in loadCfg
    raise Exception
Exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\__init__.py", line 65, in run_keyword
    return DynamicCore.run_keyword(self, name, args, kwargs)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\robotlibcore.py", line 173, in run_keyword
    return self.keywords[name](*args, **(kwargs or {}))
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 85, in testsuite_setup
    raise Exception(TM.CTestsuitesCfg.oConfig.sLoadedCfgLog['unknown'])

[HolQue]

2024-11-12 14:22:58.011812 - FAIL - Unable to load the test configuration. The test execution will be aborted!
2024-11-12 14:22:58.011812 - DEBUG - Traceback (most recent call last):
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\__init__.py", line 65, in run_keyword
    return DynamicCore.run_keyword(self, name, args, kwargs)
  File "C:\workplace\RobotFramework\python39\lib\site-packages\robotlibcore.py", line 173, in run_keyword
    return self.keywords[name](*args, **(kwargs or {}))
  File "C:\workplace\RobotFramework\python39\lib\site-packages\RobotFramework_TestsuitesManagement\Keywords\CSetup.py", line 96, in testsuite_setup
    raise Exception(TM.CTestsuitesCfg.oConfig.sLoadedCfgLog['unknown'])

[test-fullautomation]

This is essential for the "original" based release.

[namsonx]

Hello Holger,

I pushed a change to stabi branch to disable traceback log in case loading configuration failed. The log of these cases is: Original

==============================================================================
2024-11-21 16:29:39.510997 - INFO - + START SUITE: Tsm-Testfile-02
==============================================================================
2024-11-21 16:29:40.076290 - INFO - +- START SETUP: tm.Testsuite Setup [ ./config/tsm-test_variants.jsonp ]
2024-11-21 16:29:40.447348 - ERROR - Required parameter 'WelcomeString' is missing in file 'C:/Users/mas2hc/Desktop/data_storage/workspace/ROBFWAIO-OSS/robotframework-testsuitesmanagement/test/testfiles/config/tsm-test_config_missing_param.jsonp'.
2024-11-21 16:29:40.449303 - ERROR - JSON schema validation failed!
2024-11-21 16:29:40.451304 - FAIL - Unable to load the test configuration. The test execution will be aborted!
2024-11-21 16:29:40.454303 - DEBUG - jsonschema.exceptions.ValidationError: 'WelcomeString' is a required property

Failed validating 'required' in schema:
    {'$id': './Config/configuration_schema.json',
     '$schema': 'http://json-schema.org/draft-07/schema#',
     'additionalProperties': False,
     'description': 'ROBFW AIO configuration file in json format',
     'properties': {'Maximum_version': {'type': ['string', 'null']},
                    'Minimum_version': {'type': ['string', 'null']},
                    'Project': {'type': 'string'},
                    'TargetName': {'type': 'string'},
                    'WelcomeString': {'type': 'string'},
                    'params': {'additionalProperties': False,
                               'properties': {'global': {}},
                               'required': ['global'],
                               'type': 'object'},
                    'version': {'properties': {'majorversion': {'type': 'string'},
                                               'minorversion': {'type': 'string'},
                                               'patchversion': {'type': 'string'}},
                                'type': 'object'}},
     'required': ['Project', 'WelcomeString', 'TargetName'],
     'title': 'robot_config',
     'type': 'object'}

On instance:
    {'Maximum_version': '1.0.0',
     'Minimum_version': '0.6.0',
     'Project': 'tsm-test_config_missing_param',
     'TargetName': 'Device_01',
     'params': {'global': {'teststring_bench': 'I am the teststring '
                                               'containing the default '
                                               'value for all test benches',
                           'teststring_common': 'I am the common '
                                                'teststring valid for all '
                                                'variants and all test '
                                                'benches',
                           'teststring_variant': "I am the 'missing_param' "
                                                 'configuration of '
                                                 'tsm-test'}}}

During handling of the above exception, another exception occurred:

Exception

During handling of the above exception, another exception occurred:

Exception: Unable to load the test configuration. The test execution will be aborted!
2024-11-21 16:29:40.454303 - INFO - +- END SETUP: tm.Testsuite Setup (0.378013 s)
------------------------------------------------------------------------------
2024-11-21 16:29:40.455303 - INFO - +- START TEST: Test Case tsm-testfile-02
------------------------------------------------------------------------------
2024-11-21 16:29:40.460333 - INFO - +- END TEST: Test Case tsm-testfile-02 (0.00503 s)
------------------------------------------------------------------------------
2024-11-21 16:29:40.464305 - INFO - +- START TEARDOWN: tm.Testsuite Teardown [ ]
2024-11-21 16:29:40.465303 - INFO - testsuite_teardown: Will be implemented later
2024-11-21 16:29:40.465303 - INFO - +- END TEARDOWN: tm.Testsuite Teardown (0.000998 s)
==============================================================================
2024-11-21 16:29:40.465303 - INFO - + END SUITE: Tsm-Testfile-02 (0.954306 s)
==============================================================================

Extended

==============================================================================
20241121 16:34:27.753 - INFO - + START SUITE: Tsm-Testfile-02 [ ]
==============================================================================
20241121 16:34:27.766 - INFO - +- START SETUP: tm.Testsuite Setup [ ./config/tsm-test_variants.jsonp ]
20241121 16:34:28.142 - ERROR - Required parameter 'WelcomeString' is missing in file 'C:/Users/mas2hc/Desktop/data_storage/workspace/ROBFWAIO-OSS/robotframework-testsuitesmanagement/test/testfiles/config/tsm-test_config_missing_param.jsonp'.
20241121 16:34:28.143 - ERROR - JSON schema validation failed!
20241121 16:34:28.144 - UNKNOWN - Unable to load the test configuration. The test execution will be aborted!
20241121 16:34:28.148 - DEBUG - jsonschema.exceptions.ValidationError: 'WelcomeString' is a required property

Failed validating 'required' in schema:
    {'$id': './Config/configuration_schema.json',
     '$schema': 'http://json-schema.org/draft-07/schema#',
     'additionalProperties': False,
     'description': 'ROBFW AIO configuration file in json format',
     'properties': {'Maximum_version': {'type': ['string', 'null']},
                    'Minimum_version': {'type': ['string', 'null']},
                    'Project': {'type': 'string'},
                    'TargetName': {'type': 'string'},
                    'WelcomeString': {'type': 'string'},
                    'params': {'additionalProperties': False,
                               'properties': {'global': {}},
                               'required': ['global'],
                               'type': 'object'},
                    'version': {'properties': {'majorversion': {'type': 'string'},
                                               'minorversion': {'type': 'string'},
                                               'patchversion': {'type': 'string'}},
                                'type': 'object'}},
     'required': ['Project', 'WelcomeString', 'TargetName'],
     'title': 'robot_config',
     'type': 'object'}

On instance:
    {'Maximum_version': '1.0.0',
     'Minimum_version': '0.6.0',
     'Project': 'tsm-test_config_missing_param',
     'TargetName': 'Device_01',
     'params': {'global': {'teststring_bench': 'I am the teststring '
                                               'containing the default '
                                               'value for all test benches',
                           'teststring_common': 'I am the common '
                                                'teststring valid for all '
                                                'variants and all test '
                                                'benches',
                           'teststring_variant': "I am the 'missing_param' "
                                                 'configuration of '
                                                 'tsm-test'}}}

During handling of the above exception, another exception occurred:

Exception

During handling of the above exception, another exception occurred:

Exception: Unable to load the test configuration. The test execution will be aborted!
20241121 16:34:28.148 - INFO - +- END SETUP: tm.Testsuite Setup (383)
------------------------------------------------------------------------------
20241121 16:34:28.150 - INFO - +- START TEST: Test Case tsm-testfile-02 [ ]
------------------------------------------------------------------------------
20241121 16:34:28.156 - INFO - +- END TEST: Test Case tsm-testfile-02 (3)
------------------------------------------------------------------------------
20241121 16:34:28.158 - INFO - +- START TEARDOWN: tm.Testsuite Teardown [ ]
20241121 16:34:28.158 - INFO - testsuite_teardown: Will be implemented later
20241121 16:34:28.158 - INFO - +- END TEARDOWN: tm.Testsuite Teardown (0)
==============================================================================
20241121 16:34:28.162 - INFO - + END SUITE: Tsm-Testfile-02 (905)
==============================================================================

Thank you, Son

[HolQue]

Hi Son,

very good. Especially I like the idea to keep the detailed jsonschema.exceptions.ValidationError information at DEBUG level. This is really helpful. I made some tries with log level USER and DEBUG in extended version. It works. With log level INFO the details are not logged any more.

Then I switched to the original version. Now I am confused. Because I see the same jsonschema.exceptions.ValidationError also in case I run the test with log level INFO. I took a look at my listings in this issue. Already there all traceback output is marked as DEBUG. I completely overlooked this. Why is a DEBUG message printed in log level USER? I tried some other log levels in command line. There are no differences in the log output. The Robot Framework does completely not react on any log level provided in command line. Except I use an invalid log level. This causes an error. I have no idea what is going one here.

OK, two things:

1. My guess is that these effects have something to do with the different Robot Framework versions behind the original and the extended version. @namsonx : Would be great if you can spend some time to try to reproduce my log level findings w.r.t the original version.
2. In my opinion the issue is solved in the Testsuites Management. Therefore the issue can be closed.

Further measures w.r.t. the different versions of the Robot Framework core should be handled in other issues (if required).