Possibility to dump all configuration parameters

HolQue commented 7 months ago

It would be really helpful for testing to have the possibility to dump the entire configuration taken out of JSON configuration files.

Especially because of JSON configuration files can be nested and values can be overwritten (either like wanted or accidently).

But trying

log dictionary ${CONFIG}

leads into an empty 'params' dictionary:

params: {}

This is really unfavorable.

I would like to have access to the params dictionary - for two reasons:

Dump all values (for documentation, test and debugging purposes)
Iterate the keys or the values to search during the test dynamically for certain information

namsonx commented 3 weeks ago

Hello @test-fullautomation,

We decided to set parameters in params[global] to global variable of RobotFramework AIO, then delete params[global] before. Now, with the request from Holger, we will keep all parameters in params[global]. And users can access these parameters by log dictionary ${CONFIG}[params][global]. Could you please confirm this change?

Thank you, Son

namsonx commented 3 weeks ago

Hello Holger,

I pushed the commit 4d4d0d39c4 to stabi branch for this ticket, all configuration parameters are visible in ${CONFIG}[params].

Thank you, Son

namsonx commented 3 weeks ago

Ready for sprint 0.13.1 [06.09.24](jsonpreprocessor / tsm)

test-fullautomation commented 2 weeks ago

Hi @namsonx, please revert this change. I also removed the "ready for testing state".

Here is a misunderstanding.

This requires some discussion and is a bit more complex feature.

Goal is that we can log configuration data before a test is started that we know the conditions for the test run as part of the log.

TML implements for this also a black- and white-list which allows filtering. Question is also how to log the data (presentation in log file(s))

Let us talk about this till we have a common understanding. Thank you, Thomas

HolQue commented 2 weeks ago

Hi Son, hi Thomas,

In general it works!

And there are some aspects that I want to discuss:

This is my configuration:

"params" : {
             "global": {
                        ${params.global.testdict.subkey1.subkey2.subkey3} : "subkeyvalue"

Now some working combinations of robot code and corresponding log file entry:

log    ${testdict}
INFO - {'subkey1': {'subkey2': {'subkey3': 'subkeyvalue'}}}

log    ${testdict}[subkey1]
INFO - {'subkey2': {'subkey3': 'subkeyvalue'}}

Interesting, it's also possible to start from top:

log    ${CONFIG}
INFO - {'WelcomeString': 'Hello... ROBFW (testversion) [SelfTest] is running now!', 'Maximum_version': '1.0.0', 'Minimum_version': '0.6.0', 'Project': 'test', 'TargetName': 'test', 'params': {'global': {'testdict': {'subkey1': {'subkey2': {'subkey3': 'subkeyvalue'}}}}}}

log    ${CONFIG}[params][global]
INFO - {'testdict': {'subkey1': {'subkey2': {'subkey3': 'subkeyvalue'}}}}

log    ${CONFIG}[params][global][testdict]
INFO - {'subkey1': {'subkey2': {'subkey3': 'subkeyvalue'}}}

[1]

What does this mean that both is possible: ${testdict} and ${CONFIG}[params][global][testdict]? Does this mean that testdict is stored twice in RAM? Or is there a mapping implemented anyhow? In my opinion it would be an unfavorable RAM consumption to store every user defined parameter twice.

[2]

Some tries with dot notation:

log    ${testdict.subkey1}
INFO - {'subkey2': {'subkey3': 'subkeyvalue'}}

log    ${CONFIG.params.global.testdict}
Resolving variable '${CONFIG.params.global.testdict}' failed: SyntaxError: invalid syntax (<string>, line 1)

Why is the second one a syntax error? In standard notation it works.

[3]

How to give guidance to the users? We tell them that they get access to their test dictionary with: ${testdict} and we tell that it's not necessary to do it in the long winded way: ${CONFIG}[params][global][testdict].

But when they want to get all user defined parameters, they need to do this: ${CONFIG}[params][global]. This is not nice, because we give the users the responsibility for knowing this internal structure: CONFIG/params/global.

In my opinion we need a keyword for this:

@keyword
def get_user_configuration(self):
   user_config = dict(BuiltIn().get_variable_value('${CONFIG}[params][global]'))
   return user_config

This would be the test code:

${user_config}    Get User Configuration
log    ${user_config}

And this the result:

INFO - {'testdict': {'subkey1': {'subkey2': {'subkey3': 'subkeyvalue'}}}}

This is in my opinion much more plausible for users.

We need a RobotFramwework AIO internal keyword library for that. I propose that we introduce such a library. Or do we have this already?

test-fullautomation commented 2 weeks ago

Hi @HolQue , please refer to https://github.com/test-fullautomation/RobotFramework_AIO/issues/235#issuecomment-2343278947 Thank you, Thomas

HolQue commented 2 weeks ago

In case you want to have all :-)

@keyword
def dump_robot_settings(self):
    # with from robot.conf import RobotSettings
    robot_settings = RobotSettings()
    return robot_settings

${robot_settings}    Dump Robot Settings
rf.extensions.pretty_print    ${robot_settings}    [robot_settings]

INFO - [robot_settings] : [<class 'robot.conf.settings.RobotSettings'>]  :  'ConsoleColors: AUTO
ConsoleMarkers: AUTO
ConsoleType: verbose
ConsoleTypeDotted: False
ConsoleTypeQuiet: False
ConsoleWidth: 78
DebugFile: None
Doc: None
DryRun: False
Exclude: []
ExitOnError: False
ExitOnFailure: False
ExpandKeywords: []
Extension: ('.robot', '.rbt', '.robot.rst')
FlattenKeywords: []
Include: []
Language: []
Listeners: []
Log: log.html
LogLevel: INFO
LogTitle: None
MaxAssignLength: 200
MaxErrorLines: 40
Metadata: []
Name: None
Output: output.xml
OutputDir: C:\workplace\ROBFW\TestTestcases\RobotFramework
ParseInclude: []
Parsers: []
PreRebotModifiers: []
PreRunModifiers: []
PythonPath: []
RPA: None
Randomize: NONE
ReRunFailed: None
ReRunFailedSuites: None
RemoveKeywords: []
Report: report.html
ReportBackground: ('#9e9', '#f66', '#66c7ff', '#fed84f')
ReportTitle: None
RunEmptySuite: False
SetTag: []
Skip: []
SkipOnFailure: []
SkipTeardownOnExit: False
SplitLog: False
StatusRC: True
StdErr: None
StdOut: None
SuiteNames: []
SuiteStatLevel: -1
TagDoc: []
TagStatCombine: []
TagStatExclude: []
TagStatInclude: []
TagStatLink: []
TaskNames: []
TestNames: []
TimestampOutputs: False
VariableFiles: []
Variables: []
VisibleLogLevel: INFO
XUnit: None'

test-fullautomation commented 2 weeks ago

Interesting information. Can be maybe part of the dump. Especially with black- white-listing interesting.

HolQue commented 2 weeks ago

W.r.t. representation in log files, here is my proposal:

Define some config values:

"params" : {
             "global": {
                        "user param 1"     : "user param value 1",
                        "user param 02"    : "user param value 02",
                        "user param 003"   : "user param value 003",
                        "user param 0004"  : "user param value 004",
                        "user param 00005" : "user param value 00005"

Within

RobotFramework_TestsuitesManagement\Keywords\CSetup.py

at end of

def testsuite_setup(self, sTestsuiteCfgFile=''):

add the following code:

user_config = BuiltIn().get_variable_value('${CONFIG}[params][global]')
BuiltIn().log(f"<font face=\"Arial\" color=\"#FF0000\" size=\"3\"><b>User configuration:</b></font>", level="INFO", html=True, console=False)
for key, value in user_config.items():
    BuiltIn().log(f"<font face=\"Arial\" color=\"#00008B\" size=\"3\"><b>{key}<font color=\"#000000\"> = </font> <i><font color=\"#006400\">{value}</font></i></b></font>", level="INFO", html=True, console=False)

Execute a test and open an HTML log file, section "Testsuite Setup". At end you will find the user configuration, formatted in HTML.

Addendum: Dump of suite configuration added:

# --------------------------------------------------------------------------------------------------------------
# ------ condiguration dump extension
# -- 1. Robot Framework configuration
# with: from robot.conf import RobotSettings
robot_settings = RobotSettings()
suite_config = robot_settings.suite_config
BuiltIn().log(f"<font face=\"Arial\" color=\"#FF0000\" size=\"3\"><b>Robot Framework configuration:</b></font>", level="INFO", html=True, console=False)
for key, value in suite_config.items():
    BuiltIn().log(f"<font face=\"Arial\" color=\"#00008B\" size=\"3\"><b>{key}<font color=\"#000000\"> = </font> <i><font color=\"#006400\">{value}</font></i></b></font>", level="INFO", html=True, console=False)
#
# -- 2. Testsuite configuration
user_config = BuiltIn().get_variable_value('${CONFIG}[params][global]')
BuiltIn().log(f"<font face=\"Arial\" color=\"#FF0000\" size=\"3\"><b>Testsuite configuration:</b></font>", level="INFO", html=True, console=False)
for key, value in user_config.items():
    BuiltIn().log(f"<font face=\"Arial\" color=\"#00008B\" size=\"3\"><b>{key}<font color=\"#000000\"> = </font> <i><font color=\"#006400\">{value}</font></i></b></font>", level="INFO", html=True, console=False)
# --------------------------------------------------------------------------------------------------------------

HolQue commented 2 weeks ago

Update:

Within RobotFramework_TestsuitesManagement\Keywords\CSetup.py

at end of

def testsuite_setup(self, sTestsuiteCfgFile=''):

add the following code:

# --------------------------------------------------------------------------------------------------------------
# ------ configuration dump extension
#
# with: from robot.conf import RobotSettings
# with: from tabulate import tabulate
#
# -- levels description
levels_info = {1 : "configuration file in command line",
               2 : "variant name in command line",
               3 : "configuration file in local config folder",
               4 : "default configuration (fallback solution)"}
#
table_rows = []
#
# -- 1. Common information (small extract from what is available in RobotSettings())
robot_settings = RobotSettings()
log_level = robot_settings.log_level
table_rows.append(["[COMMON]", "log level", "=", f"{log_level}"])
output = robot_settings.output
table_rows.append(["[COMMON]", "output", "=", f"{output}"])
log = robot_settings.log
table_rows.append(["[COMMON]", "log", "=", f"{log}"])
report = robot_settings.report
table_rows.append(["[COMMON]", "report", "=", f"{report}"])
split_log = robot_settings.split_log
table_rows.append(["[COMMON]", "split log", "=", f"{split_log}"])
include = robot_settings.include
table_rows.append(["[COMMON]", "include", "=", f"{include}"])
exclude = robot_settings.exclude
table_rows.append(["[COMMON]", "exclude", "=", f"{exclude}"])
pythonpath = robot_settings.pythonpath
table_rows.append(["[COMMON]", "pythonpath", "=", f"{pythonpath}"])
status_rc = robot_settings.status_rc
table_rows.append(["[COMMON]", "status_rc", "=", f"{status_rc}"])
remove_keywords = robot_settings.remove_keywords
table_rows.append(["[COMMON]", "remove_keywords", "=", f"{remove_keywords}"])
#
# -- 2. Testsuite setup
current_level_number = TM.CTestsuitesCfg.oConfig.configLevel.value
current_level_info   = levels_info[current_level_number]
table_rows.append(["[TESTSUITE SETUP]", "configuration level", "=", f"{current_level_number} ({current_level_info})"])
table_rows.append(["[TESTSUITE SETUP]", "number of test suites", "=", f"{TM.CTestsuitesCfg.oConfig.iSuiteCount}"])
table_rows.append(["[TESTSUITE SETUP]", "number of testcases", "=", f"{TM.CTestsuitesCfg.oConfig.iTotalTestcases}"])
#
# -- 3. Meta data
suite_metadata = BuiltIn().get_variables()['&{SUITE_METADATA}']
for key, value in suite_metadata.items():
    table_rows.append(["[METADATA]", f"{key}", "=", f"{value}"])
#
# -- 4. Testsuite configuration
user_config = BuiltIn().get_variable_value('${CONFIG}[params][global]')
max_char = 200
for key, value in user_config.items():
    str_value = f"{value}"
    if len(str_value) > max_char:
        value = str_value[:max_char] + " ..."
    table_rows.append(["[TESTSUITE CONFIG]", f"{key}", "=", f"{value}"])
#
# -- convert to table and log
parameter_table = tabulate(table_rows, tablefmt="fancy_grid")
BuiltIn().log("\n" + parameter_table, level="INFO", html=False, console=True)
# ------------------------------------------------------------------

test-fullautomation commented 2 weeks ago

Hi @namsonx , I talked already with Holger about the table representation. Please implement according to Holger's proposal.

Next week we need to define how we do black- and whitelisting that we log only the wanted data.

Thank you, Thomas

HolQue commented 2 weeks ago

Hi Thomas, hi Son,

independent from this table solution I think, my questions in

https://github.com/test-fullautomation/RobotFramework_AIO/issues/235#issuecomment-2343280070

are still valid. By occasion it would be helpful to have a clarification w.r.t. that.

namsonx commented 2 weeks ago

Hello Thomas, hello Holger,

A console log with table data is attached in this ticket. datatable_log.log Can you please help me review and give your opinion?

Thank you, Son

HolQue commented 2 weeks ago

Hi Son,

looks very impressive!

namsonx commented 1 week ago

Hello Thomas, Hello Holger,

I created keyword Dump Config Info to log out configuration parameters in html format and return user configuration object. html_log.zip Can you please help me review and give your opinion?

Thank you, Son

test-fullautomation / RobotFramework_AIO

Possibility to dump all configuration parameters #235