Inconsistent error message (installation path)

HolQue commented 1 year ago

In package_context.json I made a key invalid:

"installer_location_invalid" : "https://github.com/test-fullautomation/RobotFramework_AIO/releases",
"bundle_name"        : "RobotFramework AIO",
"bundle_version"     : "0.8.0.0",
"bundle_version_date": "05.2023"

Outcome is an inconsistent error message:

The test execution requires maximum RobotFramework AIO version '0.2.0'
but the installed RobotFramework AIO version is younger        '0.8.0.0'

Please install the required RobotFramework AIO version.
You can find an installer here: https://github.com/test-fullautomation/robotframework-testsuitesmanagement/releases

The content of package_context.json should be checked more strict.

HolQue commented 1 year ago

The same with other keys.

"bundle_version_invalid" : "0.8.0.0",

causes a wrong version (the one from testsuites management) used for version check:

The test execution requires maximum RobotFramework AIO version '0.2.0'
but the installed RobotFramework AIO version is younger        '0.7.1'

test-fullautomation commented 1 year ago

@namsonx : What is the status of this issue?

namsonx commented 1 year ago

Hello @ngoan1608, can you update the status of this ticket?

test-fullautomation commented 1 year ago

solved and already merged with #220

HolQue commented 1 year ago

Retest result:

Every invalid key causes a corresponding error message now. OK.

But missing keys are not considered.

Usecase:

Version check fail, therefore "installer_location" and "bundle_name" required for error message.
In case of "installer_location" and "bundle_name" are missing in file package_context.json, the version check error message contains installer_location and bundle_name from TestsuitesManagement (instead of the one from RobotFramework AIO)

In my opinion the file package_context.json should also be checked for completeness.

HolQue commented 1 week ago

The behavior changed (probably caused by 0.13.1 changes).

In case of an invalid "installer_location" key, a lot of stuff is written to console:

[ ERROR ] Invalid '... \site-packages\RobotFramework_TestsuitesManagement\Config\package_context.json' file. Reason: Additional properties are not allowed ('installer_location_invalid' was unexpected)

Failed validating 'additionalProperties' in schema:
    {'type': 'object',
     'additionalProperties': False,
     'properties': {'installer_location': {'type': 'string'},
                    'bundle_name': {'type': 'string'},
                    'bundle_version': {'type': 'string'},
                    'bundle_version_date': {'type': 'string'}}}

On instance:
    {'installer_location_invalid': 'https://github.com/test-fullautomation/RobotFramework_AIO/releases',
     'bundle_name': 'RobotFramework AIO extended',
     'bundle_version': '0.13.1.6',
     'bundle_version_date': '11.2024'}
[ ERROR ] Execution stopped by user.
------------------------------
RF returned ERRORLEVEL : 253

And the debug log file is mostly empty:

20241120 18:43:34.420 - INFO - + START SUITE: RobotFramework [ ]

test-fullautomation commented 1 week ago

Hi @namsonx , this bundle version control is one of our main features. Therefore this is Prio1 for 0.13.1. Please check. Thank you, Thomas

HolQue commented 1 week ago

https://github.com/test-fullautomation/robotframework-testsuitesmanagement/issues/191#issuecomment-2489215231 solved in between.

No, I have to correct myself: not solved. By mistake I tested with standard JSONP configuration file. But this issue is about the package_context.json file. Now I faced: The behavior of this file completely different.

Test ongoing.

HolQue commented 1 week ago

Restest with package_context.json / extended version / level INFO

invalid key (= additional property)

"installer_location_invalid" : "https://github.com/test-fullautomation/RobotFramework_AIO/releases",

Console:

[ ERROR ] Invalid '... \Config\package_context.json' file. Reason: Additional properties are not allowed ('installer_location_invalid' was unexpected) (and additional details)

Debug log file:

completely empty

syntax error (missing comma)

Console:

[ ERROR ] Cannot load the '... \Config\package_context.json' file. Reason: Expecting ',' delimiter: line 3 column 3 (char 98)
[ ERROR ] Execution stopped by user.

Debug log file:

completely empty

missing key ("installer_location" removed)

Is ignored (maybe because of this information is not used because the version check is passed). Test is passed.

missing key ("installer_location" removed) together with failed version check

[ ERROR ] Wrong use of max/min version control in configuration.
Testsuite : C:\workplace\ROBFW\TestTestcases\RobotFramework\quicktest.robot
config    : C:/workplace/ROBFW/TestTestcases/RobotFramework/config/testconfig.json

The configured minimum RobotFramework AIO extended version                 '1.6.0'
is younger than the configured maximum RobotFramework AIO extended version '1.0.0'
Please correct the values of 'Maximum_version', 'Minimum_version' in config file

Please install the required RobotFramework AIO extended version.
You can find an installer here: https://github.com/test-fullautomation/robotframework-testsuitesmanagement/releases

QuickTest :: QuickTest documentation                                  | UNKNOWN |

The provided installer location is wrong (belongs to component, but not to bundle). Therefore It's better to check package_context.json for completeness at first.

missing key ("bundle_name" removed) together with failed version check

Same issue as in item before:

Please install the required RobotFramework_TestsuitesManagement version.
You can find an installer here: https://github.com/test-fullautomation/RobotFramework_AIO/releases

Bundle name does not fit to installer path.

test-fullautomation / robotframework-testsuitesmanagement

Inconsistent error message (installation path) #191