Infinity errors when using together custom required argument option and xdist parallelization

[RockBomber]

Hi All!

I have a simple test with custom required argument option:

# conftest.py

def pytest_addoption(parser):
    parser.addoption('--required-option', required=True)

# test_1.py

def test_1():
    pass

When I run tests in single process mode, I have no errors. But when I run tests in parallelization mode with xdist, I've got infinity errors:

$ pytest --required-option 2 -n 2
================================================================================= test session starts ==================================================================================
platform linux2 -- Python 2.7.8, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
rootdir: /home/asdef/Dropbox/Work/Git/PyTestError, inifile: 
plugins: bdd-2.17.1, html-1.10.1, xdist-1.15.0
gw0 C / gw1 Cusage: -c [options] [file_or_dir] [file_or_dir] [...]
-c: error: argument --required-option is required
usage: -c [options] [file_or_dir] [file_or_dir] [...]
-c: error: argument --required-option is required
[gw0] node down: Traceback (most recent call last):
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 1072, in executetask
    do_exec(co, loc)  # noqa
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 155, in <module>
  File "<remote exec>", line 129, in remote_initconfig
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 912, in fromdictargs
    config.parse(args, addopts=False)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 1040, in parse
    args = self._parser.parse_setoption(args, self.option, namespace=self.option)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 508, in parse_setoption
    parsedoption = self.parse(args, namespace=namespace)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 489, in parse
    return self.optparser.parse_args([str(x) for x in args], namespace=namespace)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 739, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1726, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1950, in _parse_known_args
    self.error(_('argument %s is required') % name)
  File "/usr/lib/python2.7/argparse.py", line 2367, in error
    self.exit(2, _('%s: error: %s\n') % (self.prog, message))
  File "/usr/lib/python2.7/argparse.py", line 2355, in exit
    _sys.exit(status)
SystemExit: 2

Replacing crashed slave gw0
gw2 C / gw1 C[gw1] node down: Traceback (most recent call last):
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 1072, in executetask
    do_exec(co, loc)  # noqa
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 155, in <module>
  File "<remote exec>", line 129, in remote_initconfig
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 912, in fromdictargs
    config.parse(args, addopts=False)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 1040, in parse
    args = self._parser.parse_setoption(args, self.option, namespace=self.option)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 508, in parse_setoption
    parsedoption = self.parse(args, namespace=namespace)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 489, in parse
    return self.optparser.parse_args([str(x) for x in args], namespace=namespace)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 739, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1726, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1950, in _parse_known_args
    self.error(_('argument %s is required') % name)
  File "/usr/lib/python2.7/argparse.py", line 2367, in error
    self.exit(2, _('%s: error: %s\n') % (self.prog, message))
  File "/usr/lib/python2.7/argparse.py", line 2355, in exit
    _sys.exit(status)
SystemExit: 2

Replacing crashed slave gw1
gw2 C / gw3 Cusage: -c [options] [file_or_dir] [file_or_dir] [...]
-c: error: argument --required-option is required
usage: -c [options] [file_or_dir] [file_or_dir] [...]
-c: error: argument --required-option is required
[gw2] node down: Traceback (most recent call last):
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 1072, in executetask
    do_exec(co, loc)  # noqa
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 155, in <module>
  File "<remote exec>", line 129, in remote_initconfig
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 912, in fromdictargs
    config.parse(args, addopts=False)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 1040, in parse
    args = self._parser.parse_setoption(args, self.option, namespace=self.option)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 508, in parse_setoption
    parsedoption = self.parse(args, namespace=namespace)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 489, in parse
    return self.optparser.parse_args([str(x) for x in args], namespace=namespace)
  File "/home/asdef/.virtualenvs/autotesting/local/lib/python2.7/site-packages/_pytest/config.py", line 739, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1726, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1950, in _parse_known_args
    self.error(_('argument %s is required') % name)
  File "/usr/lib/python2.7/argparse.py", line 2367, in error
    self.exit(2, _('%s: error: %s\n') % (self.prog, message))
  File "/usr/lib/python2.7/argparse.py", line 2355, in exit
    _sys.exit(status)
SystemExit: 2

...

[RonnyPfannschmidt]

this is a bug in the configuration initialization of pytest core, its incredibly hard to fix and related to #1618

[BrandonBlair]

Having this issue as well, but the referenced ticket won't load for me. You mention it's "incredibly hard to fix", are there any other resources that may help me address it?

[RonnyPfannschmidt]

oh, github broke on that one :-1:

the first step is to remove the required flag for the argument and check in a pytest-configure hook its a crude hack but works

a "good" fix needs larger refactoring in both pytest and xdist, i didnt even find time to investigate the exact needs yet

[BrandonBlair]

Awesome, thanks. I pass environment in as a required argument, and I don't like using a default as the user could potentially think they're testing something they're not.

[RonnyPfannschmidt]

use none as the default and fail in pytest_configure with a pytest.UsageError

[ultinous-meli2200]

Is there any fix for this the bug? I have same problem. My platform : linux -- Python 3.5.2, pytest-3.5.0, py-1.5.3, pluggy-0.6.0 plugins: xdist-1.22.2, forked-0.2

[RonnyPfannschmidt]

@ultinous-meli2200 no there isnt, please use the mentioned workaround