dynamically added value to addopts with pytest_configure is not reflecting in cmd line

[GitBharath]

Objective:

Adding -n [var_num] to addopts for my pytest.ini file wherein the var_num is dynamically taken accordingly from command line for pytest-xdist run.

python3.6, pytest==4.6.3, pytest-xdist==1.28.0

My project struct:

Python-package
    - pytest.ini
    - setup.py
    - src (py package)
         - conftest.py
         - config.py
         - tests
             - test_tc1.py
             - test_tc2.py

pytest.ini

[pytest]
addopts = -v -s -p no:warnings
filterwarnings =
    ignore::UserWarning

conftest.py

def pytest_configure(config):
    config.addinivalue_line("markers", "env(name): mark test to run only on named environment")
    count = len(config.getoption('--env'))
    print('count: : ', count)
    args = "-n" + str(count)
    print('args is: ', args)
    config.addinivalue_line("addopts", args)
    print(config.getini("addopts"))

output is

(venv) [root@localhost tests]$ pytest test_tc1.py --env ENV1 --env ENV2 --env ENV3
count: :  3
args is:  -n3
['-v', '-s', '-p', 'no:warnings', '-n3']
===================================================================== test session starts ======================================================================
platform linux -- Python 3.6.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- [ROOT_DIR]/venv/bin/python3.6
cachedir: .pytest_cache
rootdir: [ROOT_DIR],
inifile: pytest.ini
plugins: forked-1.0.2, rerunfailures-7.0, xdist-1.28.0
collecting ... [shortened data]
collected 3 items                                                                                                                                              

test_tc1.py::TestClass1::test_tc1[ENV1] 
[Shortened Data]
PASSED[Shortened Data]

test_tc1.py::TestClass1::test_tc1[ENV2] 
[Shortened Data]
PASSED[Shortened Data]

test_tc1.py::TestClass1::test_tc1[ENV3] 
[Shortened Data]
PASSED[Shortened Data]

================================================================== 3 passed in 10.49 seconds ===================================================================

Expected Output

Expectation was should be same as when I parse the -n flag to pytest.ini

pytest.ini

[pytest]
addopts = -v -s -p no:warnings -n3
filterwarnings =
    ignore::UserWarning

conftest.py

def pytest_configure(config):
    config.addinivalue_line("markers", "env(name): mark test to run only on named environment")
    print(config.getini("addopts"))

Result is shown as below

(venv) root@localhost tests]$ pytest test_tc1.py --env ENV1 --env ENV2 --env ENV3
['-v', '-s', '-p', 'no:warnings', '-n3']
===================================================================== test session starts ======================================================================
platform linux -- Python 3.6.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- [ROOT_DIR]/venv/bin/python3.6
cachedir: .pytest_cache
rootdir: [ROOT_DIR], inifile: pytest.ini
plugins: forked-1.0.2, rerunfailures-7.0, xdist-1.28.0
[gw0] linux Python 3.6.3 cwd: [TEST_DIR]/tests
[gw1] linux Python 3.6.3 cwd: [TEST_DIR]/tests
[gw2] linux Python 3.6.3 cwd: [TEST_DIR]/tests
[gw0] Python 3.6.3 (default, Jun 25 2019, 18:53:50)  -- [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
[gw1] Python 3.6.3 (default, Jun 25 2019, 18:53:50)  -- [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
[gw2] Python 3.6.3 (default, Jun 25 2019, 18:53:50)  -- [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
gw0 ok / gw1 ok / gw2 ok/[Shortened Data]
gw0 [3] / gw1 [3] / gw2 [3]
scheduling tests via LoadScheduling

test_tc1.py::TestClass1::test_tc1[ENV3] 
test_tc1.py::TestClass1::test_tc1[ENV2] 
test_tc1.py::TestClass1::test_tc1[ENV1] 
[gw0] PASSED  test_tc1.py::TestClass1::test_tc1[ENV1] 
[gw2] PASSED  test_tc1.py::TestClass1::test_tc1[ENV3] 
[gw1] PASSED  test_tc1.py::TestClass1::test_tc1[ENV2] 

=================================================================== 3 passed in 6.05 seconds ===================================================================

As you see even if both the pytest.ini addopts having the same list of data, the resulting behaviour is different. The first attempt is not even considering the flag -n. What is the reason for this ? How to get this fixed ?

EDIT

If I add both the pytest.ini and conftest.py files; the list data is

['-v', '-s', '-p', 'no:warnings', '-n3', '-n3']

[GitBharath]

@kevlened @nicodemus

[GitBharath]

@nicoddemus

[nicoddemus]

Hi,

Sorry I'm in a hurry so I don't have time to look at the code of pytest-parallel in detail:

1. Why is -n3 being passed here, given that -n is used by pytest-xdist?
2. Does the behavior change when run with or without pytest-parallel?
3. I don't think changing addopts in pytest_configure will have any effect, regardless of pytest-parallel; by the time pytest_configure has been called, the arguments have already been processed by pytest.

[GitBharath]

Why is -n3 being passed here, given that -n is used by pytest-xdist?

I parsed -n3 so that the xdist runs on 3 subprocesses

Does the behavior change when run with or without pytest-parallel?

No. But If you have time; please look into the other bug #34

I don't think changing addopts in pytest_configure will have any effect, regardless of pytest-parallel; by the time pytest_configure has been called, the arguments have already been processed by pytest

I need to somehow configure the addopts either in command line or pytest.ini.

[nicoddemus]

I parsed -n3 so that the xdist runs on 3 subprocesses

I'm a little confused because this is the issue tracker for pytest-parallel... does pytest-parallel uses pytest-xdist somehow?

No. But If you have time; please look into the other bug #34

Sorry, I fail to see the connection with what's being discussed here. 😬

I need to somehow configure the addopts either in command line or pytest.ini.

Doesn't really make sense to configure addopts from the command-line, given that addopts is to implicitly add options to the command-line. addopts in pytest.ini is just a matter of writing the options there...

Can you explain exactly what you are trying to accomplish (rather than how)?

[GitBharath]

I'm a little confused because this is the issue tracker for pytest-parallel... does pytest-parallel uses pytest-xdist somehow?

yes it uses pytest-xdist that is the reason I am giving the flag -n here

Sorry, I fail to see the connection with what's being discussed here.

There is no connection, please look into it sometime later, let's discuss on the current issue

Can you explain exactly what you are trying to accomplish (rather than how)?

My attempt is to append the flag -n=[num_of_environments_parsed_in_cmd_line] to pytest.ini dynamically with hooks or custom plugins. Have tried with several hooks like pytest_cmdline_main, pytest_cmdline_preparse, pytest_cmdline_parse and pytest_configure. but unable to get what I wanted.

In order to resolve the issue, I have gone through the issues and comments raised by some people with similar attempts in bugs "https://github.com/pytest-dev/pytest/issues/3302" and "https://github.com/pytest-dev/pytest/issues/1150". But no help found with the comments given.

[nicoddemus]

Thanks! OK, now things are starting to click together for me. 👍

It does not seem to be possible to actually change the command-line arguments via a plugin, because it is processed very early in the initialization process.

You can however change the option directly in a hook:

# conftest.py
def pytest_cmdline_preparse(config, args):
    config.option.numprocesses = 2

pytest-xdist will check config.option.numprocesses during pytest_cmdline_main:

https://github.com/pytest-dev/pytest-xdist/blob/9f9b70748115cf0da7ae06aa1fb1127b6ba757a9/src/xdist/plugin.py#L166-L175

[GitBharath]

Ok thanks for the answer. I was just having a weird thought that if we somehow put the command line to a mediator test data before processing/do initial configuring immediately to run the scripts. And configuration can happen according to the parsed test data.

Command line --> test data  --> *.ini/.cfg files --> test scripts run

Not sure if this is a good idea. Maybe we can close this off.

But do let me know your comments. Thanks.

[nicoddemus]

I was just having a weird thought that if we somehow put the command line to a mediator test data before processing/do initial configuring immediately to run the scripts. And configuration can happen according to the parsed test data.

Currently this is tricky, because command-options can be used to control plugin loading, for example -p.

[kevlened]

I'm a little confused because this is the issue tracker for pytest-parallel... does pytest-parallel uses pytest-xdist somehow?

yes it uses pytest-xdist that is the reason I am giving the flag -n here

pytest-parallel doesn't use pytest-xdist; it monkey-patches pytest with multiprocessing and thread-safe data structures.

[marcobazzani]

Thanks! OK, now things are starting to click together for me. 👍

It does not seem to be possible to actually change the command-line arguments via a plugin, because it is processed very early in the initialization process.

You can however change the option directly in a hook:

# conftest.py
def pytest_cmdline_preparse(config, args):
    config.option.numprocesses = 2

pytest-xdist will check config.option.numprocesses during pytest_cmdline_main:

https://github.com/pytest-dev/pytest-xdist/blob/9f9b70748115cf0da7ae06aa1fb1127b6ba757a9/src/xdist/plugin.py#L166-L175

you saved my day works on xdist too