kondratyev-nv / vscode-python-test-adapter

Python Test Adapter for the VS Code Test Explorer
https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
MIT License
117 stars 27 forks source link

Pytest's Xfail as Skipped and Xpass as Passed in Test Explorer UI #276

Open domeniqo opened 2 years ago

domeniqo commented 2 years ago

If I get it right, this adapter evaluates results on generated xml file from pytest.

Is there possibility to add other icon in test explorer UI than is used for "skipped" tests? It would be great to distinguish between expected failures and skipped tests.

XFail: Pytest's generated xml file has node \<skipped type="pytest.xfail" \/> which could be parsed during tests result evaluation.

XPass: Pytest's generated xml file does not determine between python.pass and pytest.xpass so this can't be probably implemented right away.

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite name="pytest" errors="0" failures="0" skipped="1" tests="2" time="2.831" timestamp="2021-11-29T12:10:05.163599" hostname="PC">
         <!-- The following test was marked with @pytest.mark.xfail and failed -->
        <testcase classname="tests.test_pytest_demo.TestsDemo" name="test_xx_ui_fail" time="0.217">
            <skipped type="pytest.xfail" message="" />
        </testcase>
        <!-- The following test was marked with @pytest.mark.xfail and passed -->
        <testcase classname="tests.test_pytest_demo.TestsDemo" name="test_xx_ui_pass" time="0.000" />
    </testsuite>
</testsuites>

Results shown in UI: image

karolzlot commented 2 years ago

"pytest.xpass" should be treated similarly to "failed" -> test passed, but it wasn't excepted to pass, something is wrong

Currently vscode treats it as "passed" which is incorrect

edit: I was wrong, it is necessary to turn on "strict" mode for this