Make function-scope fixtures run in the same thread as the test function itself

[ogrisel]

Currently, pytest fixtures seem to always be executed in the main thread. This prevents initializing thread-local configuration to run a particular test in isolation of concurrently running tests with different thread-local settings.

See the reproducer below:

# test_simple.py
import threading

import pytest

thread_local = threading.local()

@pytest.fixture
def change_thread_local():
    print("inside fixture", threading.current_thread())
    setattr(thread_local, "value", 1)
    yield

def test_simple(change_thread_local):
    print("inside function", threading.current_thread())
    assert getattr(thread_local, "value", None) == 1

pytest --threads 1 --iterations 1 test_simple.py

resulting in:

============================================================================================================= test session starts ==============================================================================================================
platform linux -- Python 3.13.0rc3, pytest-8.3.3, pluggy-1.5.0
rootdir: /tmp/proj
plugins: xdist-3.6.1, freethreaded-0.1.0
collected 1 item                                                                                                                                                                                                                               

test_simple.py F                                                                                                                                                                                                                         [100%]

=================================================================================================================== FAILURES ===================================================================================================================
_________________________________________________________________________________________________________________ test_simple __________________________________________________________________________________________________________________

change_thread_local = None

    def test_simple(change_thread_local):
        print("inside function", threading.current_thread())
>       assert getattr(thread_local, "value", None) == 1
E       AssertionError: assert None == 1
E        +  where None = getattr(<_thread._local object at 0x7b3843ff32e0>, 'value', None)

test_simple.py:17: AssertionError
------------------------------------------------------------------------------------------------------------ Captured stdout setup -------------------------------------------------------------------------------------------------------------
inside fixture <_MainThread(MainThread, started 135481613739584)>
------------------------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------------------------
inside function <Thread(ThreadPoolExecutor-0_0, started 135481585043136)>
=========================================================================================================== short test summary info ============================================================================================================
FAILED test_simple.py::test_simple - AssertionError: assert None == 1
============================================================================================================== 1 failed in 0.03s ===============================================================================================================

Would it be possible to execute function-scope fixtures on the same thread as the test function?

Originally discussed in the context of testing scikit-learn in free threading mode: https://github.com/scikit-learn/scikit-learn/issues/30007

[lesteve]

I bumped into pytest-parallel that has been archived and is no longer maintained (latest release in October 2021 and see https://github.com/kevlened/pytest-parallel/issues/104#issuecomment-1293941066), but seems to run the fixture and the test function in the same thread. The following command runs fine:

pytest --tests-per-worker 2 test_simple.py

See their plugin code.

Full disclosure, I had to edit the plugin code to avoid an AttributeError (probably something has changed in pytest since):

self._log = print # py.log.Producer('pytest-parallel')

[ogrisel]

As discussed in the linked scikit-learn issue, it's possible to convert such thread-sensitive fixtures to regular function decorators and the problems go away.

So we have a workaround.

[ogrisel]

Note that the above workaround is not always valid. For instance, the standard tmpdir fixture does no longer ensure working in an isolated temporary directory for a parametrized test:

@pytest.mark.parametrize("value", range(10))
def test_stuff_in_isolated(value, tmpdir):
    subfolder = tmpdir.mkdir("subfoler")  # fails with py.error.EEXIST: [File exists]...
    # Do test stuff in "subfolder" and "value".