box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
381 stars 59 forks source link

passing test fails if run in Windows on GH actions #185

Open qci-amos opened 2 years ago

qci-amos commented 2 years ago

I have a set of tests I run with pytest in my Windows build in on GH Actions.

I originally wrote this up as a pytest issue: https://github.com/pytest-dev/pytest/discussions/9321 you can see some discussion there about the perspective from pytest.

but now it looks like it's a flaky issue since if I skip the test that's been marked with flaky, then the test passes.

from that issue above, the stdout for my pytest looks like this:

============================= test session starts =============================
platform win32 -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
***dir: D:\a\<snip>
plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, mock-3.6.1, timeout-2.0.1
collected 32 items

test\test_utils.py ..                                                    [  6%]
test\test_client.py ......ss..ss..sss............s                       [100%]

============================== warnings summary ===============================
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\stringcase-1.2.0-py3.8.egg\stringcase.py:247
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\stringcase-1.2.0-py3.8.egg\stringcase.py:247
  c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\stringcase-1.2.0-py3.8.egg\stringcase.py:247: DeprecationWarning: invalid escape sequence \W

<snip>

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===Flaky Test Report===

test_cancel[Signals.CTRL_C_EVENT] passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
=========================== short test summary info ===========================
SKIPPED [8] test\conftest.py:146: Set DUMMY_JCS to the name of an available JCS to test
============ 24 passed, 8 skipped, 6 warnings in 447.29s (0:07:27) ============
Error: Process completed with exit code 1.
martijnvangils commented 2 years ago

Hi, did you solve this issue yet? Curious for the solution

qci-amos commented 2 years ago

I don't know how to solve this issue, so I just skipped it:

    if sys.platform == "win32":
        pytest.skip("this test doesn't work on GH actions on Windows?")