Race condition in test_python_to_cpp_to_python_from_process test on 3.9?

[YannickJadoul]

Not sure what's going on, but this test failed again on 3.9, completely unrelated to the actual changes. I don't know what that test is about, but I thought I'd log it.

Do we have a race condition? Does CPython 3.9 have a race condition?

__________________ test_python_to_cpp_to_python_from_process ___________________

    def test_python_to_cpp_to_python_from_process():
        """Makes sure there is no GIL deadlock when using processes.

        This test is for completion, but it was never an issue.
        """
>       assert _run_in_process(_python_to_cpp_to_python) == 0
E       assert -11 == 0
E        +  where -11 = _run_in_process(_python_to_cpp_to_python)

test_gil_scoped.py:81: AssertionError

[henryiii]

We also see a segfault here occasionally for Python 3.9:

-------------------------------------------------------------------------------
Restart the interpreter
-------------------------------------------------------------------------------
/Users/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:108
...............................................................................

/Users/runner/work/pybind11/pybind11/tests/test_embed/test_interpreter.cpp:108: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal

===============================================================================
test cases:  4 |  3 passed | 1 failed
assertions: 16 | 15 passed | 1 failed

make[3]: *** [tests/test_embed/CMakeFiles/cpptest] Error 139
make[2]: *** [tests/test_embed/CMakeFiles/cpptest.dir/all] Error 2
make[1]: *** [tests/test_embed/CMakeFiles/cpptest.dir/rule] Error 2

[EricCousineau-TRI]

Also in test_python_to_cpp_to_python_from_thread_multiple_parallel: https://github.com/pybind/pybind11/pull/2396#issuecomment-674267074

[henryiii]

Is this mostly macOS? I could add an xfail(env.MACOS and env.pyv >= 3.9, reason="Occasional failure, TBD", strict=False) here, to help with unrelated PRs, but a little worried that might cause us to overlook it until Python 3.9 is released and it's too late to fix it if it's on Python's end.

[henryiii]

test_python_to_cpp_to_python_from_thread_multiple_sequential too. Seems to always be macOS.

[YannickJadoul]

Anything on this? Is this still happening? If so, we should probably report this to Python before the final release is out.

[YannickJadoul]

Another instantiation of this fluke?

``` _______________________ test_multiple_inheritance_python _______________________ @pytest.mark.skipif("env.PYPY and env.PY2") @pytest.mark.xfail("env.PYPY and not env.PY2") def test_multiple_inheritance_python(): class MI1(m.Base1, m.Base2): def __init__(self, i, j): m.Base1.__init__(self, i) m.Base2.__init__(self, j) class B1(object): def v(self): return 1 class MI2(B1, m.Base1, m.Base2): def __init__(self, i, j): B1.__init__(self) m.Base1.__init__(self, i) m.Base2.__init__(self, j) class MI3(MI2): def __init__(self, i, j): MI2.__init__(self, i, j) class MI4(MI3, m.Base2): def __init__(self, i, j): MI3.__init__(self, i, j) # This should be ignored (Base2 is already initialized via MI2): m.Base2.__init__(self, i + 100) class MI5(m.Base2, B1, m.Base1): def __init__(self, i, j): B1.__init__(self) m.Base1.__init__(self, i) m.Base2.__init__(self, j) class MI6(m.Base2, B1): def __init__(self, i): m.Base2.__init__(self, i) B1.__init__(self) class B2(B1): def v(self): return 2 class B3(object): def v(self): return 3 class B4(B3, B2): def v(self): return 4 class MI7(B4, MI6): def __init__(self, i): B4.__init__(self) MI6.__init__(self, i) class MI8(MI6, B3): def __init__(self, i): MI6.__init__(self, i) B3.__init__(self) class MI8b(B3, MI6): def __init__(self, i): B3.__init__(self) MI6.__init__(self, i) mi1 = MI1(1, 2) assert mi1.foo() == 1 assert mi1.bar() == 2 mi2 = MI2(3, 4) assert mi2.v() == 1 assert mi2.foo() == 3 assert mi2.bar() == 4 > mi3 = MI3(5, 6) ../../tests/test_multiple_inheritance.py:133: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../tests/test_multiple_inheritance.py:78: in __init__ MI2.__init__(self, i, j) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = .MI3 object at 0x7f6a95e1f0e0> i = 5, j = 6 def __init__(self, i, j): B1.__init__(self) > m.Base1.__init__(self, i) E TypeError: __init__(self, ...) called with invalid `self` argument ../../tests/test_multiple_inheritance.py:73: TypeError ```

EDIT: This is not, though. This is related to #2564

[henryiii]

Running 3.9 macOS locally, the tests pass, but I get a segfault popup once in a while:

0   org.python.python               0x000000010bf4f34a meth_dealloc + 120
1   org.python.python               0x000000010bf2145a instancemethod_dealloc + 65
2   org.python.python               0x000000010bf43feb PyDict_Clear + 409
3   org.python.python               0x000000010bf616ec type_clear + 59
4   org.python.python               0x000000010c00ef00 collect + 2558
5   org.python.python               0x000000010c00e4f1 _PyGC_CollectNoFail + 67
6   org.python.python               0x000000010bfe119c _PyImport_Cleanup + 1545
7   org.python.python               0x000000010bff1680 Py_FinalizeEx + 160
8   org.python.python               0x000000010bff2167 Py_Exit + 13
9   org.python.python               0x000000010bff8117 handle_system_exit + 35
10  org.python.python               0x000000010bff676f _PyErr_PrintEx + 40
11  org.python.python               0x000000010bff64e8 PyRun_SimpleStringFlags + 116
12  org.python.python               0x000000010c00d3c7 Py_RunMain + 425
13  org.python.python               0x000000010c00dc86 pymain_main + 306
14  org.python.python               0x000000010c00dcd4 Py_BytesMain + 42
15  libdyld.dylib                   0x00007fff6910dcc9 start + 1

This happens when one of the XFails fails instead of XPasses.

[YannickJadoul]

Running 3.9 macOS locally, the tests pass, but I get a segfault popup once in a while:

This is another one, though, it seems. This is described in #2558 and debugged (and hopefully soon resolved) in #2576. Doesn't seem to be related to the issue described here, though.

[YannickJadoul]

More info in #2754