tests: fix random segfault on GNU/Linux

BoboTiG commented 1 year ago

Such a segfault happens quite often, event after:

247
248
249

$ xvfb-run python -m pytest
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib
============================= test session starts ==============================
platform linux -- Python 3.12.0a7, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/python-mss/python-mss
configfile: setup.cfg
plugins: cov-4.0.0
collecting ... collected 77 items

src/tests/test_bgra_to_rgb.py::test_bad_length PASSED                    [  1%]
src/tests/test_bgra_to_rgb.py::test_good_types PASSED                    [  2%]
src/tests/test_cls_image.py::test_custom_cls_image PASSED                [  3%]
src/tests/test_find_monitors.py::test_get_monitors PASSED                [  5%]
src/tests/test_find_monitors.py::test_keys_aio PASSED                    [  6%]
src/tests/test_find_monitors.py::test_keys_monitor_1 PASSED              [  7%]
src/tests/test_find_monitors.py::test_dimensions PASSED                  [  9%]
src/tests/test_get_pixels.py::test_grab_monitor PASSED                   [ 10%]
src/tests/test_get_pixels.py::test_grab_part_of_screen PASSED            [ 11%]
src/tests/test_get_pixels.py::test_grab_part_of_screen_rounded PASSED    [ 12%]
src/tests/test_get_pixels.py::test_grab_individual_pixels PASSED         [ 14%]
src/tests/test_gnu_linux.py::test_factory_systems PASSED                 [ 15%]
src/tests/test_gnu_linux.py::test_arg_display PASSED                     [ 16%]
src/tests/test_gnu_linux.py::test_bad_display_structure PASSED           [ 18%]
src/tests/test_gnu_linux.py::test_no_xlib_library PASSED                 [ 19%]
src/tests/test_gnu_linux.py::test_no_xrandr_extension PASSED             [ 20%]
src/tests/test_gnu_linux.py::test_xrandr_extension_exists_but_is_not_enabled PASSED [ 22%]
src/tests/test_gnu_linux.py::test_unsupported_depth PASSED               [ 23%]
src/tests/test_gnu_linux.py::test_region_out_of_monitor_bounds PASSED    [ 24%]
src/tests/test_gnu_linux.py::test__is_extension_enabled_unknown_name PASSED [ 25%]
src/tests/test_gnu_linux.py::test_missing_fast_function_for_monitor_details_retrieval PASSED [ 27%]
src/tests/test_gnu_linux.py::test_with_cursor PASSED                     [ 28%]
src/tests/test_gnu_linux.py::test_with_cursor_but_not_xfixes_extension_found PASSED [ 29%]
src/tests/test_gnu_linux.py::test_with_cursor_failure PASSED             [ 31%]
src/tests/test_implementation.py::test_incomplete_class[MSS0] PASSED     [ 32%]
src/tests/test_implementation.py::test_incomplete_class[MSS1] PASSED     [ 33%]
src/tests/test_implementation.py::test_incomplete_class[MSS2] PASSED     [ 35%]
src/tests/test_implementation.py::test_bad_monitor PASSED                [ 36%]
src/tests/test_implementation.py::test_repr PASSED                       [ 37%]
src/tests/test_implementation.py::test_factory PASSED                    [ 38%]
src/tests/test_implementation.py::test_entry_point[False] PASSED         [ 40%]
src/tests/test_implementation.py::test_entry_point[True] PASSED          [ 41%]
src/tests/test_implementation.py::test_entry_point_error[False] PASSED   [ 42%]
src/tests/test_implementation.py::test_entry_point_error[True] PASSED    [ 44%]
src/tests/test_implementation.py::test_grab_with_tuple PASSED            [ 45%]
src/tests/test_implementation.py::test_grab_with_tuple_percents PASSED   [ 46%]
src/tests/test_implementation.py::test_thread_safety Fatal Python error: Segmentation fault

Thread 0x00007f4dad7d1640 (most recent call first):
  File "/home/runner/work/python-mss/python-mss/src/mss/linux.py", line 431 in _grab_impl
  File "/home/runner/work/python-mss/python-mss/src/mss/base.py", line 90 in grab
  File "/home/runner/work/python-mss/python-mss/src/tests/test_implementation.py", line 227 in record
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 989 in run
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 1052 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 1009 in _bootstrap

Current thread 0x00007f4dae7d2640 (most recent call first):
  File "/home/runner/work/python-mss/python-mss/src/mss/linux.py", line 319 in __init__
  File "/home/runner/work/python-mss/python-mss/src/mss/factory.py", line 34 in mss
  File "/home/runner/work/python-mss/python-mss/src/tests/test_implementation.py", line 226 in record
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 989 in run
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 1052 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 1009 in _bootstrap

Thread 0x00007f4db33d8b80 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 1146 in _wait_for_tstate_lock
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/threading.py", line 1126 in join
  File "/home/runner/work/python-mss/python-mss/src/tests/test_implementation.py", line 239 in test_thread_safety
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/python.py", line 1799 in runtest
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/main.py", line 323 in _main
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 166 in main
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.7/x64/lib/python3.12/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, PIL._imaging (total: 14)
Segmentation fault (core dumped)
Error: Process completed with exit code 139.

On the other hand, #247 did a good job at preventing more segfaults, but we now have more failures like:

 _________________________ test_grab_individual_pixels __________________________

    def test_grab_individual_pixels():
        monitor = {"top": 160, "left": 160, "width": 222, "height": 42}
>       with mss(display=os.getenv("DISPLAY")) as sct:

monitor    = {'height': 42, 'left': 160, 'top': 160, 'width': 222}

src/tests/test_get_pixels.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/mss/factory.py:34: in mss
    return linux.MSS(**kwargs)
        kwargs     = {'display': ':99'}
        linux      = <module 'mss.linux' from '/home/runner/work/python-mss/python-mss/src/mss/linux.py'>
        os_        = 'linux'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <mss.linux.MSS object at 0x7fe4caad0b80>, kwargs = {'display': ':99'}
display = b':99'

    def __init__(self, /, **kwargs: Any) -> None:
        """GNU/Linux initialisations."""

        super().__init__(**kwargs)

        # Available thread-specific variables
        self._handles = local()
        self._handles.display = None
        self._handles.drawable = None
        self._handles.original_error_handler = None
        self._handles.root = None

        display = kwargs.get("display", b"")
        if not display:
            try:
                display = os.environ["DISPLAY"].encode("utf-8")
            except KeyError:
                raise ScreenShotError("$DISPLAY not set.") from None

        if not isinstance(display, bytes):
            display = display.encode("utf-8")

        if b":" not in display:
            raise ScreenShotError(f"Bad display value: {display!r}.")

        if not _X11:
            raise ScreenShotError("No X11 library found.")
        self.xlib = cdll.LoadLibrary(_X11)

        if not _XRANDR:
            raise ScreenShotError("No Xrandr extension found.")
        self.xrandr = cdll.LoadLibrary(_XRANDR)

        if self.with_cursor:
            if _XFIXES:
                self.xfixes = cdll.LoadLibrary(_XFIXES)
            else:
                self.with_cursor = False

        self._set_cfunctions()

        # Install the error handler to prevent interpreter crashes: any error will raise a ScreenShotError exception
        self._handles.original_error_handler = self.xlib.XSetErrorHandler(_error_handler)

        self._handles.display = self.xlib.XOpenDisplay(display)
        if not self._handles.display:
>           raise ScreenShotError(f"Unable to open display: {display!r}.")
E           mss.exception.ScreenShotError: Unable to open display: b':99'.

__class__  = <class 'mss.linux.MSS'>
display    = b':99'
kwargs     = {'display': ':99'}
self       = <mss.linux.MSS object at 0x7fe4caad0b80>

src/mss/linux.py:321: ScreenShotError

Lets see if we can improve the situation.

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

BoboTiG commented 1 year ago

Note: it targets all Python versions (from 3.8 to 3.12 at the moment), and implementations (CPython, and PyPy).

BoboTiG commented 1 year ago

About the failure, it seems that only test_grab_individual_pixels() has the issue. It just happened again on CPython 3.10 (job)

BoboTiG commented 1 year ago

test_grab_individual_pixels() was renamed test_get_pixel() in #252. no more failure is expected given that it no more uses MSS to grad a screenshot.

But I enhances test_grab_part_of_screen() at the same time to take a lot more region screenshots. If something must fail, it will be there :)

BoboTiG commented 1 year ago

I'll merge #252, it almost never passes on the CI (either we hit the segfault, or the failure). @mgorny how do you feel about that? Do you think we can manage to find a fix a some point? 🤞🏻

BoboTiG commented 1 year ago

Nevermind, I simplified the test to no trigger the issue. But now, the same failure happens with other tests like test_keys_aio(). There is definitively something wrong with the CI. I can't reproduce locally.

mgorny commented 1 year ago

Hmm but test_keys_aio() doesn't spawn another Xvfb but uses the one spawned by GHA, right? This is weird.

I suppose we could just start using pytest-rerunfailures to have the tests retry a few times if necessary. That's just a hack, though but it should help distinguish real problems from flakiness. I'll make a PR shortly, I have a high system load right now, so I can reproduce ;-).

BoboTiG commented 1 year ago

Hmm but test_keys_aio() doesn't spawn another Xvfb but uses the one spawned by GHA, right? This is weird.

Indeed. I think a test run before might break something. Or xvfb-run may have hard time for whatever reason.

I suppose we could just start using pytest-rerunfailures to have the tests retry a few times if necessary. That's just a hack, though but it should help distinguish real problems from flakiness. I'll make a PR shortly, I have a high system load right now, so I can reproduce ;-).

Great add 👍🏻 Thanks !

BoboTiG / python-mss

tests: fix random segfault on GNU/Linux #251

247

248

249

Upvote & Fund