search

allankp / pytest-testrail

pytest plugin for integration with TestRail, for creating testruns and updating results
MIT License
96 stars 124 forks source link

test results from pytest.skip getting posted to TestRail as "Blocked" #83

Closed reverbc closed 5 years ago

reverbc commented 5 years ago

Is your feature request related to a problem? Please describe.

By current implementation, if a test has been marked as skipped (by @pytest.mark.skip marker), then it won't get posted to TestRail -> showing Untested on TestRail

However, if the skip happens inside test method (by pytest.skip(...) function), then it will still get posted to TestRail -> showing Blocked on TestRail.

For some of the tests I'm working on are using multiple @pytest.mark.parametrize(...) markers to test in different combinations, and there might be a combination that is actually good case and I want to skip. The test results from all of the combinations by design will report to the same test case in the same test run (which is awesome), but currently if there's one skipped combination, the whole test case will be marked as Blocked, which is preventing real status being showed.

Describe the solution you'd like

Apply same behavior (not report to TestRail) for both @pytest.mark.skip marker and pytest.skip(...) function.

Describe alternatives you've considered

Make it configurable - let user to decide whether to mark the skipped scenario as Blocked or Untested (don't report to TestRail at all).

Additional context

I'm happy to work on a PR if this suggestion is approved. Thanks!

allankp commented 5 years ago

@reverbc These seems like a realisable request, I would suggest just marking the test as 'Untested' as that is the actual state of the test. Adding an option to an option to mess with a test result seems like the wrong thing to do. Fire up a PR, happy to help / review.

reverbc commented 5 years ago

Thanks for the feedback @allankp . However, according to API doc we can not update Untested to TestRail when reporting test results: image

My way to workaround this restriction is to always include all test cases from selected test suite, and don't report those skipped tests (whether from marker or from function inside test) to TestRail. If in a test run no test result being reported to one test case, then it will automatically in Untested status.

Not sure my use case fulfill others' needs, but if it does I can work on a PR to not to report skipped test results to TestRail.

allankp commented 5 years ago

No activity