Using --argumentfile causes tests to be run in a sequence X times per process even though --testlevelsplit is used

[jkeset]

Issue:

During the execution of rfbrowser tests in the provided example, it's apparent that with the given command input, you should be capable of running three tests concurrently, each in a distinct browser. This functionality operates correctly without the --argumentfile option, as expected. However, upon using it, the tests are instead executed sequentially, resulting in each browser being opened three times. For instance, if three processes are initiated, each one will execute Test 1. Once that's done, the same will repeat for Test 2... etc

Expected behaviour?:

When utilizing --testlevelsplit alongside --argumentfile, the expected behavior is for the execution to correctly manage three distinct browsers concurrently, as specified per test case.

cmd input: pabot --testlevelsplit --processes 3 --argumentfile someargumentfile.robot samplerobotfile.robot

pip libraries and resources: robotframework==7.0 robotframework-browser==18.1.0 robotframework-pabot==2.18.0

python version == 3.12.2 node version == v20.9.0

Sample code:

*** Settings ***

Library    Browser

*** Test Cases ***
Test 1
    [Tags]    some_tag
    [Template]      Test templates for opening browser ${browser_name}  
    chromium

Test 2
    [Tags]    some_tag
    [Template]      Test templates for opening browser ${browser_name}  
    firefox

Test 3
    [Tags]    some_tag
    [Template]      Test templates for opening browser ${browser_name}  
    webkit

*** Keywords ***

Test templates for opening browser ${browser_name}
    Open Browser    https://www.google.com    ${browser_name}
    Go to    https://www.amazon.com
    Go to    https://www.ebay.com
    Go to    https://www.facebook.com
    Go to    https://www.yahoo.com
    Log    ${browser_name}

Sample argument file --include some_tag

[mkorpela]

Thanks for the report. This is related to recent change on Robot Framework core project. Making include and test flags incremental. There is ongoing discussion on what should be done next (should this be reverted on Robot Framework side or fixed here). One other related issue is https://github.com/robotframework/robotframework/issues/5023