Unfavorable error messages in RobotLog2DB

[HolQue]

(1)

Within a Robot Framework result XML file I set the value of all status nodes to value "WHATEVER".

Outcome is the following error message:

Successfully connected to: ...
Created test execution result for version '20230203_120839' successfully: 7b0b3850-6536-4460-8c33-100d94a70f8a
  Created test file result for file 'C:\RobotTest\projects\ROBFW_AIO\robotframework-extensions-collection\test\01.test.pretty_print.robot' successfully: 14821
ERROR: Invalid Robotframework result state 'WHATEVER' of test 'PrettyPrintTest_1'.
ERROR: Invalid Robotframework result state 'WHATEVER' of test 'PrettyPrintTest_2'.
  Created test file result for file 'C:\RobotTest\projects\ROBFW_AIO\robotframework-extensions-collection\test\02.test.normalize_path.robot' successfully: 14822
ERROR: Invalid Robotframework result state 'WHATEVER' of test 'NormalizePathTest_1'.
ERROR: Invalid Robotframework result state 'WHATEVER' of test 'NormalizePathTest_2'.
ERROR: Invalid Robotframework result state 'WHATEVER' of test 'NormalizePathTest_3'.
All test results written to database successfully.

Here I have doubts: Even in case of invalid values are detected (all status values are invalid!) the feedback is, that test file results are created successfully and that all results are written to database.

I would prefer to have the complete content checked before a connection to the database is established and before the app starts writing values to database. This would prevent us from dealing with incomplete or invalid test results in database.

And the app return value in this case is: 0. But because of errors happened I would prefer to have an individual error code.

(2)

Within a Robot Framework result XML file I removed all status nodes.

Outcome is the following error message:

Successfully connected to: ...
FATAL ERROR: Could not create new execution result. Reason: expected string or bytes-like object
C:/RobotTest/projects/ROBFW_AIO/robotframework-robotlog2db/RobotLog2DB/robotlog2db.py has been stopped!

This does not help because does not point to the root cause: a missing tag.

It is possible to detect missing tags. Therefore I would expect an error message telling me the name of what is missed instead of giving hints about the expected data type.

In this context I am also confused to read this: "Could not create new execution result". This seems to be a very app internal thing. Does this part of the error message contain anything useful for the users? I don't think so. And the wording is mistakable. Because new execution results will not be created. They were already available, in the result file. The test creates execution results, not the database app. What - maybe - the message wants to tell is, that because of missing input values no new entries in the database were made.

In case you want to stay on this part in the error message, you should rephrase it.

[test-fullautomation]

Generic solution for all *db tools which use XML as input is

1. validate input XML against XML schema.
2. generate proper error message in case of violation against schema.
3. lxml as dependency provides corresponding functionality

[HolQue]

Maybe this can be used:

https://github.com/robotframework/robotframework/tree/master/doc/schema

[HolQue]

Incompatible:

https://github.com/robotframework/robotframework/blob/master/doc/schema/robot.xsd is already adapted to Robot Framework 5.0 with schema version 3 (whereas our current Robot Framework version 4.1.3 is still on schema version 2.

Merge effort:

Official Robot Framework schema does not contain our UNKNOWN state. This must be added (after every update).

Unfavorable error message:

After setting the schema version manually back to 2 (just a try) and after adding UNKNOWN to the schema file I simulated an error by validating a pytest XML result file with the Robot Framework schema (assuming that by mistake a user selects a wrong XML result file). This is the error message:

Element 'testsuites': No matching global declaration available for the validation root., line 1!

This is much technical and does not help the users.

Therefore I propose to check the root tag only (and manually in the code, without schema). The goal is to have a chance to tell a user that he has provided an XML file of wrong type. That's all we need. In case of already the root tag is not the expected one, the remaining content of the XML result file is completely irrelevant in this case and the execution of the database app can be stopped.

[test-fullautomation]

solved and merged with #36