Testing of `pytest` fails with `pytest-enabler`

[mtelka]

When there is pytest-enabler installed together with either pytest-black and/or pytest-mypy and/or pytest-cov then running test for pytest fails with errors like this one:

__________________ TestConfigFromdictargs.test_basic_behavior __________________

self = <test_config.TestConfigFromdictargs object at 0x7fffab105a60>
_sys_snapshot = None 

    def test_basic_behavior(self, _sys_snapshot) -> None:
        option_dict = {"verbose": 444, "foo": "bar", "capture": "no"}
        args = ["a", "b"]

        config = Config.fromdictargs(option_dict, args)
        with pytest.raises(AssertionError):
            config.parse(["should refuse to parse again"])
        assert config.option.verbose == 444
        assert config.option.foo == "bar"
        assert config.option.capture == "no"
>       assert config.args == args
E       AssertionError: assert ['a', 'b'] == ['a', 'b', '-...ypy', '--cov']
E         Right contains 3 more items, first extra item: '--black'
E         Full diff:
E         - ['a', 'b', '--black', '--mypy', '--cov']
E         + ['a', 'b']

testing/test_config.py:895: AssertionError

More details: https://github.com/pytest-dev/pytest/issues/11401

[jaraco]

I recommend not installing pytest-enabler except in environments where you want the effect of pytest-enabler to take place. It's primary purpose is to automatically supply defaults based on the presence of plugins. If you plan to run pytest tests in an environment with pytest-enabler and any plugins it supports, you'll probably want to disable the enabler with something like PYTEST_ADDOPTS=-p no:"plugin-enabled options".

Let me know if you think this project should be doing something differently.

[jaraco]

I don't care for the fact that the plugin must be referred as "plugin-enabled options". That probably should be just "enabler". I'm considering renaming that, but unsure when or how, so I'll await some feedback here first.

[mtelka]

The need for the -p no:XXX option to disable the pytest-enabler plugin when it is not used seems to be valid and acceptable requirement.

The only problem is that pytest-enabler is enabled by default and it needs to be manually disabled for pytest subcalls. This is how I need to do that: https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/components/python/pytest/patches/01-pytest-disable-plugins.patch. Yes, the patch contains the problem described by @The-Compiler here, but you see the idea that I simply need to patch pytest tests to make them pass. It would be great if we slowly converge to the world without such patches needed.

I think that means that either pytest-enabler should be disabled by default (I'm not sure it is possible without losing main ptest-enabler advantages), or we need a way how to pass -p no:XXX options to pytest subcalls. Currently PYTEST_ADDOPTS works for the main pytest only. For all subcalls it is simply ignored.

[mtelka]

I don't care for the fact that the plugin must be referred as "plugin-enabled options". That probably should be just "enabler". I'm considering renaming that, but unsure when or how, so I'll await some feedback here first.

Yes, "plugin-enabled options" is ugly, but I can live with it :-). Honestly, I do not care much about the exact name. I do not mind if you rename it to "enabler" or to any different name. Just please do not do the rename too often :-)).

[jaraco]

is possible without losing main pytest-enabler advantages

It's true - pytest-enabler aims for "installed or not" to be the deciding factor on whether it takes effect. My preference would be for pytest-enabler not to be installed if it's not intended to have its effect. I'd also be open to pytest-enabler honoring a "disabler", either in the CLI or the environment vars (e.g. PYTEST_ENABLER_SUPPRESSED=True), but my instinct is that PYTEST_ADDOPTS=-p no:{whatever the name is} should be sufficient.

Currently PYTEST_ADDOPTS works for the main pytest only. For all subcalls it is simply ignored.

That's unexpected. I'd like to understand why that is, as it may reveal a deficiency in pytest.

[mtelka]

Currently PYTEST_ADDOPTS works for the main pytest only. For all subcalls it is simply ignored.

That's unexpected. I'd like to understand why that is, as it may reveal a deficiency in pytest.

After thinking a bit more about this...

I think the PYTEST_ADDOPTS should not be inherited to subcalls as is, just the list of disabled plugins. IOW, when I disable a pytest plugin (either via PYTEST_ADDOPTS, or on command line, or using any other way), then such plugin should stay disabled for all runpytest() (and co) calls. If that's the case then there won't be any problem with the pytest-enabler plugin because I'll just disable it, and it will really stay disabled. So yes, this looks like a pytest flaw, not a bug in pytest-enabler.

[mtelka]

Today I played a bit with testing of pytest-xdist plugin (version 3.4.0) and its relation to other pytest plugins and I found that pytest-enabler is somehow toxic.

When I ran tests for pytest-xdist with no other pytest plugin installed, then all tests passed there. When I installed both pytest-enabler and pytest-mypy and then disabled both plugins using PYTEST_ADDOPTS="-p no:'plugin-enabled options' -p no:mypy" environment variable then tests for pytest-xdist failed:

============================= test session starts ==============================
platform sunos5 -- Python 3.9.16, pytest-7.4.3, pluggy-1.3.0 -- $(BUILD_DIR)/.tox/py39/bin/python
cachedir: .tox/py39/.pytest_cache
rootdir: $(BUILD_DIR)
configfile: tox.ini
testpaths: testing
plugins: xdist-3.4.0
collecting ... collected 203 items

testing/acceptance_test.py::TestDistribution::test_n1_pass FAILED        [  0%]

=================================== FAILURES ===================================
________________________ TestDistribution.test_n1_pass _________________________

self = <acceptance_test.TestDistribution object at 0x7fffadcd36a0>
pytester = <Pytester PosixPath('/tmp/pytest-of-marcel/pytest-208/test_n1_pass0')>

    def test_n1_pass(self, pytester: pytest.Pytester) -> None:
        p1 = pytester.makepyfile(
            """
            def test_ok():
                pass
        """
        )
        result = pytester.runpytest(p1, "-n1")
        assert result.ret == 0
>       result.stdout.fnmatch_lines(["*1 passed*"])
E       Failed: nomatch: '*1 passed*'
E           and: '============================= test session starts =============================='
E           and: 'platform sunos5 -- Python 3.9.16, pytest-7.4.3, pluggy-1.3.0'
E           and: 'rootdir: /tmp/pytest-of-marcel/pytest-208/test_n1_pass0'
E           and: 'plugins: xdist-3.4.0, mypy-0.10.3, enabler-2.3.1'
E           and: 'created: 1/1 worker'
E           and: '1 worker [3 items]'
E           and: ''
E           and: '...                                                                      [100%]'
E           and: '===================================== mypy ====================================='
E           and: 'Success: no issues found in 1 source file'
E           and: '============================== 3 passed in 8.99s ==============================='
E       remains unmatched: '*1 passed*'

$(BUILD_DIR)/testing/acceptance_test.py:22: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform sunos5 -- Python 3.9.16, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/pytest-of-marcel/pytest-208/test_n1_pass0
plugins: xdist-3.4.0, mypy-0.10.3, enabler-2.3.1
created: 1/1 worker
1 worker [3 items]

...                                                                      [100%]
===================================== mypy =====================================
Success: no issues found in 1 source file
============================== 3 passed in 8.99s ===============================

When I try with either pytest-enabler or pytest-mypy installed (and disabled) only (just single plugin at a time) then testing of pytest-xdist pass in both cases.

So it looks like that my claim above:

Currently PYTEST_ADDOPTS works for the main pytest only. For all subcalls it is simply ignored.

is not fully true. It is true only when pytest-enabler is installed. It does not matter if the pytest-enabler is enabled or not. It simply affects other plugins visibility immediately after it is installed. I do not understand how pytest-enabler is able to "activate" itself when it is disabled to mess list of available plugins, but apparently it does so somehow.

[mtelka]

FYI, I filled a bug for pluggy: https://github.com/pytest-dev/pluggy/issues/457

[jaraco]

I do not understand how pytest-enabler is able to "activate" itself when it is disabled

Me neither. This project doesn't do anything particularly idiosyncratic except how it handles enabling pytest-cov, and even then, everything must be happening through the entry point.

[jaraco]

It looks like the bug reported in pluggy fully covers the deficiency described above. I don't think there's anything else that pytest-enabler can do, but I'm open to suggestions. Feel free to revive the conversation if there's more to be done here.