do not report failures in skipped tests

[asottile-sentry]

resolves #6

[untitaker]

looks fine, can you add a test though? I think the testsuite is pretty much the only thing we have to know whether pytest upgrade would break things

[asottile-sentry]

looks fine, can you add a test though? I think the testsuite is pretty much the only thing we have to know whether pytest upgrade would break things

I couldn't get the existing testsuite running locally -- it seems to always fail?

$ pytest tests
you forgot to activate ./venv -- I gotchu
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/asottile/workspace/pytest-sentry
plugins: sentry-0.1.9
collected 13 items                                                             

tests/test_envvars.py ..                                                 [ 15%]
tests/test_fixture.py ..                                                 [ 30%]
tests/test_hub.py ...                                                    [ 53%]
tests/test_retries.py F                                                  [ 61%]
tests/test_skip.py ssXx                                                  [ 92%]
tests/test_tracing.py .E                                                 [100%]

==================================== ERRORS ====================================
_______________________ ERROR at teardown of test_basic ________________________

    @pytest.fixture(scope="session", autouse=True)
    def assert_report():
        yield
>       (event,) = events
E       ValueError: not enough values to unpack (expected 1, got 0)

tests/test_retries.py:24: ValueError
=================================== FAILURES ===================================
__________________________________ test_basic __________________________________

request = <FixtureRequest for <Function test_basic>>

    @pytest.mark.flaky(reruns=2)
    @pytest.mark.sentry_client(Client(transport=events.append, traces_sample_rate=0.0))
    def test_basic(request):
        global i
        i += 1
        if i < 2:
>           1 / 0
E           ZeroDivisionError: division by zero

tests/test_retries.py:18: ZeroDivisionError
=============================== warnings summary ===============================
tests/test_retries.py:12
  /home/asottile/workspace/pytest-sentry/tests/test_retries.py:12: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.flaky(reruns=2)

tests/test_skip.py:10
  /home/asottile/workspace/pytest-sentry/tests/test_skip.py:10: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.flaky(reruns=2)

tests/test_skip.py:16
  /home/asottile/workspace/pytest-sentry/tests/test_skip.py:16: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.flaky(reruns=2)

tests/test_skip.py:23
  /home/asottile/workspace/pytest-sentry/tests/test_skip.py:23: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.flaky(reruns=2)

tests/test_skip.py:30
  /home/asottile/workspace/pytest-sentry/tests/test_skip.py:30: PytestUnknownMarkWarning: Unknown pytest.mark.flaky - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.flaky(reruns=2)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_retries.py::test_basic - ZeroDivisionError: division by zero
ERROR tests/test_tracing.py::test_basic - ValueError: not enough values to unpack (expected 1, got 0)
= 1 failed, 8 passed, 2 skipped, 1 xfailed, 1 xpassed, 5 warnings, 1 error in 0.23s =

[untitaker]

hmm ok

can you dump pip freeze?

it does work for me (locally) after installing ./dev-requirements.txt

[untitaker]

I see you're with pytest 7, it's possible that we pytest-sentry is broken for newer pytest already

[asottile-sentry]

oh I was missing the rerunning thing :thinking: seems to pass now

[untitaker]

"somebody" (me prob) should set up CI for this. last CI here was travis at best

[asottile-sentry]

actually the current tests/test_skip.py are broken and this patch fixes them:

$ PYTEST_SENTRY_ALWAYS_REPORT=1 pytest tests/test_skip.py 
you forgot to activate ./venv -- I gotchu
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/asottile/workspace/pytest-sentry
plugins: rerunfailures-10.2, sentry-0.1.9
collected 4 items                                                              

tests/test_skip.py ssXxR                                                 [100%]xR [100%]xE [100%]

==================================== ERRORS ====================================
_______________________ ERROR at teardown of test_xfail ________________________

    @pytest.fixture(scope="session", autouse=True)
    def assert_report():
        yield
>       assert not events
E       AssertionError: assert not [{'_meta': {'exception': {'values': {'0': {'stacktrace': {...}}}}}, 'breadcrumbs': {'values': []}, 'contexts': {'runti...: 'pytest.runtest.protocol', 'parent_span_id': None, 'span_id': 'af79841e33695276', ...}}, 'environment': 'test', ...}]

tests/test_skip.py:39: AssertionError
=========================== short test summary info ============================
ERROR tests/test_skip.py::test_xfail - AssertionError: assert not [{'_meta': {'exception': {'values': {'0': {'stac...
========== 2 skipped, 3 xfailed, 1 xpassed, 1 error, 2 rerun in 0.14s ==========
$ git checkout -- .
$ PYTEST_SENTRY_ALWAYS_REPORT=1 pytest tests/test_skip.py 
you forgot to activate ./venv -- I gotchu
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/asottile/workspace/pytest-sentry
plugins: rerunfailures-10.2, sentry-0.1.9
collected 4 items                                                              

tests/test_skip.py ssXx                                                  [100%]

=================== 2 skipped, 1 xfailed, 1 xpassed in 0.08s ===================

[asottile-sentry]

I'll try and set something up :+1: