mkorpela / pabot

Parallel executor for Robot Framework test cases.
https://pabot.org
Apache License 2.0
476 stars 152 forks source link

Test Cases are not executed in expected order when using --ordering option #306

Open RiceKrisBs opened 4 years ago

RiceKrisBs commented 4 years ago

I have a repo set up that roughly has a structure similar to this:

.
├── Results
│   └──(general folder to dump results)
│
├── .pabotsuitenames-ordering
│
├── Locators
│   ├── Locator_File_1.robot
│   └── Locator_File_2.robot
│
├── Keywords
│   ├── Keyword_File_1.robot
│   └── Keyword_File_2.robot
│
└── Test_Cases
    ├── TC_Folder_1
    │   ├── Test_Case_1-1.robot
    │   └── Test_Case_1-2.robot
    └── TC_Folder_2
        ├── Test_Case_2-1.robot
        ├── Test_Case_2-2.robot
        ├── Test_Case_2-3.robot
        └── Test_Case_2-4.robot

In our implementation, the Test Case files import the Keywords files as resources and the Keyword files import the Locator files as resources. What I'd like to do is run all 4 test cases in the TC_Folder_2 directory while controlling the order using the --ordering option. I've created a .pabotsuitenames-ordering file that looks like the following:

--suite TC Folder 2.Test Case 2-1
#WAIT
{
--suite TC Folder 2.Test Case 2-4
--suite TC Folder 2.Test Case 2-2
}
--suite TC Folder 2.Test Case 2-3

I'm running the following command: pabot --ordering .\.pabotsuitenames-ordering --outputdir .\Results .\Test_Cases\TC_Folder_2

Based on my understanding from the README, this ordering file should:

  1. Execute Test_Case_2-1.robot.
  2. Wait until the above suite is complete.
  3. Start Test_Case_2-4.robot and Test_Case_2-3.robot in separate executor processes.
  4. Once Test_Case_2-4.robot is finished, start Test_Case_2-2.robot.

However, I am observing the following:

  1. Test_Case_2-1.robot is executed.
  2. It waits until the above suite is complete.
  3. Test_Case_2-2.robot and Test_Case_2-3.robot are started in separate executor processes.
  4. Once Test_Case_2-2.robot is finished, Test_Case_2-4.robot is started.

Basically, pabot does not appear to be following the defined order within the {} executor process I defined - within this executor process it's instead executing these test suites in alphabetical order. Is this expected behavior? Or is this a bug, and I should be able to control the order within an individual executor process?

dchourasia commented 4 years ago

@RiceKrisBs what pabot version you are using? I am also facing one similar issue with pabot 0.98, my all grouped tests are getting skipped, not executing at all

here is the ordering file: --test Suites.suite1.testcase 1 { --test Suites.suite1.testcase 2 --test Suites.suite2.testcase 3 } --test Suites.suite2.testcase 4

testcase 2 & 3 not executing at all.

RiceKrisBs commented 4 years ago

@RiceKrisBs what pabot version you are using? I am also facing one similar issue with pabot 0.98, my all grouped tests are getting skipped, not executing at all

I am using the latest, 1.10.0.