Quansight-Labs / pytest-run-parallel

A simple pytest plugin to run tests concurrently
MIT License
6 stars 3 forks source link

Probably parallel execution does not work with unittest #2

Closed vfdev-5 closed 2 months ago

vfdev-5 commented 3 months ago

I'm trying this tool to run tests using multi-threaded context: Here is the code I using for a check of multi-threaded execution:

import unittest

def test_failed1():
    import threading

    print("test_failed1+ident:", threading.current_thread().ident)
    assert False

class TestABC(unittest.TestCase):

    def test_failed2(self):
        import threading

        print("TestABC+test_failed2+ident:", threading.current_thread().ident)
        assert False

and execute it with the following command:

pytest --parallel-threads=3 tests/test_abc.py -v

and I see the following output:

=============================================================================================== FAILURES ===============================================================================================
_____________________________________________________________________________________________ test_failed1 _____________________________________________________________________________________________

    def test_failed1():
        import threading

        print("test_failed1+ident:", threading.current_thread().ident)
>       assert False
E       assert False

tests/test_abc.py:8: AssertionError
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
test_failed1+ident:test_failed1+ident:test_failed1+ident:  139805692208704139805700601408

139805708994112
_________________________________________________________________________________________ TestABC.test_failed2 _________________________________________________________________________________________

self = <tests.test_abc.TestABC testMethod=test_failed2>

    def test_failed2(self):
        import threading

        print("TestABC+test_failed2+ident:", threading.current_thread().ident)
>       assert False
E       assert False

tests/test_abc.py:18: AssertionError
----------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------
TestABC+test_failed2+ident: 139806255169536
======================================================================================= short test summary info ========================================================================================
FAILED tests/test_abc.py::test_failed1 - assert False
FAILED tests/test_abc.py::TestABC::test_failed2 - assert False
========================================================================================== 2 failed in 0.29s ===========================================================================================

We can see that test_failed1 printed 3 times test_failed1+ident and thread ident but the TestABC::test_failed2 printed current thread ident only once, so I supposed that it ran the test in one thread only and not 3 threads as required in the command.

@andfoy can you please check this? Thanks!

andfoy commented 3 months ago

@vfdev-5 thanks for opening this issue, simultaneous printing under free threaded Python leads to stdout to not respect printing output order, if you detail the output trace that you shared, you can see that all the output is being concatenated:

139805692208704139805700601408 139805708994112

This is something that's occurring without this plugin. From what I've experienced, this is the current behavior in CPython.

andfoy commented 3 months ago

With respect to the second test executions, it seems like a legit issue. Let me check how pytest collection works