Executions not recognized from output.xml

[manykarim]

When running sherlock --output .\TA3S-GSFS-Automation\FLOW_TESTS-final.xml .\TA3S-GSFS-Automation --report print,html,json we can successfully generate a report and a console output.

But the Executionsin the report are shown as 0 for all tests and keyword. But we can see in the .xml and the .html that it was a huge test run with hundreds of test cases.

The tests use the Datadriver to do DataDriven testing. Not sure if that might be the reason why nothing is shown.

[neiljhowell]

Similar situation here, while running sherlock -o c:\temp\output.xml c:\git\automation-framework-reponame I can successfully generate console output (haven't tried report generation yet, no reason to think it wouldn't work)

One issue in my case is that the output.xml is an aggregated xml from many of the robot suites in the repo running in parallel. I have added a few things to the sherlock source code, maybe this is causing the parsing to not line up correctly or something. Yet even it seems that keywords on the test suite page are not ticked for execution:

│ │ ├── woPhoto │ │ │ ├── ExpectedData │ │ │ └── Suite: woPhoto.robot │ │ │ │ │ │ Elapsed time │ │ │ ┌────────────────┬─────────────────────┬────────────────────┬────────────────────┐ │ │ │ │ Total elapsed │ Shortest execution │ Longest execution │ Average execution │ │ │ │ ├────────────────┼─────────────────────┼────────────────────┼────────────────────┤ │ │ │ │ 0 │ 0 │ 0 │ 0 │ │ │ │ └────────────────┴─────────────────────┴────────────────────┴────────────────────┘ │ │ │ Keywords: │ │ │ ┌────────────────────────────────┬────────────┬────────────┬───────────────┬─────────────┐ │ │ │ │ Name │ Executions │ Complexity │ Average time │ Total time │ │ │ │ ├────────────────────────────────┼────────────┼────────────┼───────────────┼─────────────┤ │ │ │ │ Call WorkOrder API Sucessfully │ 0 │ 1 │ │ │ │ │ │ │ WOPhoto Query Suite Setup │ 0 │ 2 │ │ │ │ │ │ └────────────────────────────────┴────────────┴────────────┴───────────────┴─────────────┘

[manykarim]

Now that you mention it: my output.xml was also merged from multiple smaller output files

[neiljhowell]

Also maybe worth mentioning, we are not only aggregating several xmls into a single xml output file, but also limiting the size of the report by calling: rebot --splitlog --removekeywords PASSED --removekeywords FOR --removekeywords WHILE --removekeywords WUKS (from http://robotframework.org/robotframework/2.9.1/RobotFrameworkUserGuide.html#id654 and http://robotframework.org/robotframework/2.9.1/RobotFrameworkUserGuide.html#id661)

[bhirsz]

Thanks for the information about merging, I will check how it does affect Sherlock - it could be that some information that we require is missing or modified (in that case maybe we could supply it separetly).

[bhirsz]

For Data Driven tests using external library it could be more tricky since the tests are generated by library. So it's harder to map results to the actual source code. But I will see what can be done

[neiljhowell]

@bhirsz I did have to add the following to my local sherlock to get sherlock to skip the name of the output log passed in via 'rebot -N Regression' in visitor.py:

def visit_suite(self, suite):
        self.imported_resources = OrderedDict()
        self.imported_libraries = OrderedDict()
        print(str(suite))
        if suite.source in self.resources:
            suite_resource = self.resources[suite.source]
        elif str(suite) == "Regression":
            suite_resource = ""

it does seem like this allowed sherlock to skip Regression and move on to the rest of the items in the output.xml file, but maybe I am also causing some unexpected results as well.