Skipped tests show up as failed

[marcel-landman]

testdox shows failed tests as red, but "incomplete, skipped or risky tests" as amber. Test explorer is showing these as red, making them indistinguishable from the failed tests.

[RobertOstermann]

Could you include a picture? And are you talking about the lines that are highlighted or the the tests themselves being marked as failing?

[marcel-landman]

Test Explorer: Terminal: FailedTest.php:

<?php
use PHPUnit\Framework\TestCase;
class FailedTest extends TestCase {
    function test_that_fails(){ $this->fail();}
}

RiskyIncompleteSkippedTest.php:

<?php
use PHPUnit\Framework\TestCase;
class RiskyIncompleteSkippedTest extends TestCase {
    function test_that_is_skipped(){ $this->markTestSkipped(); }
    function test_that_is_incomplete(){ $this->markTestIncomplete(); $this->assertTrue(true);}
    function test_that_is_risky_because_it_has_no_assert(){}
}

SucceedingTest.php

<?php
use PHPUnit\Framework\TestCase;
class SucceedingTest extends TestCase {
    function test_that_succeeds(){ $this->assertTrue(true); }
}

reference: https://phpunit.readthedocs.io/en/9.5/incomplete-and-skipped-tests.html

[RobertOstermann]

I could change the output so the tests are marked as skipped instead of failing. Is this the type of change you are wanting?

[marcel-landman]

Yes! Also, the folder containing risky/incomplete/skipped should show a different icon so that you can see that there are unsuccessful tests even when collapsed. I guess it could be the same icon as the worst test. It would also be good if the risky test has a more severe looking icon than skipped/incomplete because risky is not intentional as incomplete/skipped are. Perhaps an amber/yellow triangle with an exclamation point.

[RobertOstermann]

That is not possible at the moment. I am limited by the options for test results that the VSCode API supports. I could change the risky test to be marked as an error test, which is a red circle with a dot. But that icon takes precedence over the failed test. The skipped test icon is only shown when collapsed if all the tests in that folder are skipped.

I do not think I can change the icon shown on the folder level, which is based on the tests inside. If there are any error tests it will show error, then failed, then passing, and last if none of those are shown then the skipped icon is shown.

My current changes have the incomplete, risky, or skipped tests showing with the skipped icon (as in the earlier image) and I'm leaning towards keeping them like that. I don't think it makes sense to have them use the error icon (as in this image). Another option would be to leave the risky tests as failed (like they are now) so they take precedence and the folder would show as failed if there is a risky test.

[marcel-landman]

I agree. risky tests are a corner case and the skipped icon more clearly indicates what happened in the case of skipped/incomplete. Also it is more easily distinguishable from failed test compared to the error, which looks almost the same.

[RobertOstermann]

The changes have been published in v1.1.8