search

mkorpela / pabot

Parallel executor for Robot Framework test cases.
https://pabot.org
Apache License 2.0
469 stars 151 forks source link

Ordering wrong after 11th test case #493

Open Delilovic opened 2 years ago

Delilovic commented 2 years ago

Hi, when I have 10 test cases, the order in log.html is correct, but when I add the 11th test, the order is wrong. The 11th test is shown under the 1st test and the 12th test is shown under the 11th (and so on) Do you have any idea why this is or how to avoid this? It is simple to test it your self, just run pabot --testlevelsplit ./path_to_your_suite.robot on the test bellow.

correct -> normal_ordering incorrect -> 11th_test_ordering incorrect -> 11th_test_non_numeric incorrect -> 12th_test_ordering

<?xml version="1.0" encoding="UTF-8"?>
<robot generator="Rebot 5.0.1 (Python 3.9.12 on win32)" generated="20220829 15:46:19.490" rpa="false" schemaversion="3">
<suite id="s1" name="Foo" source="C:\Users\User\Development\Tester\suite\Foo.robot">
<test id="s1-t1" name="test1" line="2">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.330" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.329" endtime="20220829 15:46:18.330"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.328" endtime="20220829 15:46:18.330">Skipped with Skip keyword.</status>
</test>
<test id="s1-t2" name="test11" line="32">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:19.089" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:19.089" endtime="20220829 15:46:19.089"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:19.088" endtime="20220829 15:46:19.090">Skipped with Skip keyword.</status>
</test>
<test id="s1-t3" name="test2" line="5">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.355" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.354" endtime="20220829 15:46:18.355"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.352" endtime="20220829 15:46:18.356">Skipped with Skip keyword.</status>
</test>
<test id="s1-t4" name="test3" line="8">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.368" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.367" endtime="20220829 15:46:18.368"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.365" endtime="20220829 15:46:18.369">Skipped with Skip keyword.</status>
</test>
<test id="s1-t5" name="test4" line="11">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.321" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.321" endtime="20220829 15:46:18.322"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.319" endtime="20220829 15:46:18.322">Skipped with Skip keyword.</status>
</test>
<test id="s1-t6" name="test5" line="14">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.324" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.323" endtime="20220829 15:46:18.324"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.322" endtime="20220829 15:46:18.325">Skipped with Skip keyword.</status>
</test>
<test id="s1-t7" name="test6" line="17">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.325" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.324" endtime="20220829 15:46:18.325"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.323" endtime="20220829 15:46:18.326">Skipped with Skip keyword.</status>
</test>
<test id="s1-t8" name="test7" line="20">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.377" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.376" endtime="20220829 15:46:18.378"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.374" endtime="20220829 15:46:18.378">Skipped with Skip keyword.</status>
</test>
<test id="s1-t9" name="test8" line="23">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:18.345" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:18.345" endtime="20220829 15:46:18.346"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:18.343" endtime="20220829 15:46:18.346">Skipped with Skip keyword.</status>
</test>
<test id="s1-t10" name="test9" line="26">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:19.092" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:19.092" endtime="20220829 15:46:19.092"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:19.090" endtime="20220829 15:46:19.093">Skipped with Skip keyword.</status>
</test>
<test id="s1-t11" name="test10" line="29">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20220829 15:46:19.089" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20220829 15:46:19.088" endtime="20220829 15:46:19.089"/>
</kw>
<status status="SKIP" starttime="20220829 15:46:19.087" endtime="20220829 15:46:19.089">Skipped with Skip keyword.</status>
</test>
<doc>[https://pabot.org/?ref=log|Pabot] result from 11 executions.</doc>
<status status="SKIP" starttime="20220829 15:46:18.269" endtime="20220829 15:46:19.093"/>
</suite>
<statistics>
<total>
<stat pass="0" fail="0" skip="11">All Tests</stat>
</total>
<tag>
</tag>
<suite>
<stat pass="0" fail="0" skip="11" id="s1" name="Foo">Foo</stat>
</suite>
</statistics>
<errors>
</errors>
</robot>

Test Cases test1 Skip

test2 Skip

test3 Skip

test4 Skip

test5 Skip

test6 Skip

test7 Skip

test8 Skip

test9 Skip

test10 Skip

# Uncomment to test invalid ordering

#test11 # Skip

#testFoo # Skip

#test12 # Skip

patos98 commented 2 years ago

Hi, The problem comes from sorting the paths of the output files to be merged. The output directories of the testcases looks something like this:

├───pabot_results
│   ├───0
│   ├───1
│   ├───10
│   ├───11
│   ├───2
│   ├───3
│   ├───4
│   ├───5
│   ├───6
│   ├───7
│   ├───8
│   └───9

Each of the subfolder contains its output.xml file. The issue is that before merging the output files, pabot sorts the output file paths with the builtin sorted function which treats them as ordinary strings messing up the order of the "numbered" directories.