Tests are not correctly marked as incomplete when creating/updating snapshots

[tklie]

As the title says, tests are not correctly marked as Incomplete when creating/updating snapshots. Instead PHPUnit displays them as Errors.

PHP 8.2.4 laravel/framework 10.7.1 phpunit/phpunit 10.1.0 spatie/phpunit-snapshot-assertions 5.0.1

Manually calling $this->markTestIncomplete('Test incomplete') within a test works perfectly fine. But this packages calls this method from within the markTestIncompleteIfSnapshotsHaveChanged() method which is using an @after annotation. Exceptions thrown during this test lifecycle appear to be treated differently.

Testcases

<?php

use App\Models\User;
use Illuminate\Foundation\Testing\TestCase;
use Spatie\Snapshots\MatchesSnapshots;

class SnapshotTest extends TestCase
{
    use MatchesSnapshots;

    public function test_snapshot_matches()
    {
        $user = User::query()->findOrFail(1);

        $this->assertMatchesJsonSnapshot($user->toJson());
    }

    public function test_is_incomplete()
    {
        $this->markTestIncomplete('This test is incomplete.');
    }
}

Command

vendor/bin/phpunit -d --update-snapshots

Expected Result

PHPUnit 10.1.0 by Sebastian Bergmann and contributors.

II                                                                  2 / 2 (100%)

Time: 00:00.143, Memory: 34.00 MB

OK, but there are issues!
Tests: 2, Assertions: 3, Incomplete: 2.

Actual Result

PHPUnit 10.1.0 by Sebastian Bergmann and contributors.

EI                                                                  2 / 2 (100%)

Time: 00:00.154, Memory: 38.50 MB

ERRORS!
Tests: 2, Assertions: 3, Errors: 1, Incomplete: 1.

[timvandijck]

@tklie thank you for reporting this problem. I believe this is an issue with PHPUnit as the same issue occurs when marking tests as incomplete in an after method even without our package.

I created an issue with them: https://github.com/sebastianbergmann/phpunit/issues/5337

[mfn]

Their response was quick, see https://github.com/sebastianbergmann/phpunit/issues/5337#issuecomment-1513000069

The "after test method" is not, and was never, intended for a use case like that. By the time such methods are called, no decision on the outcome of a test can be made anymore. That this worked for you in the past, with PHPUnit 9, for instance, was accidental.

The markTestIncompleteIfSnapshotsHaveChanged() method in your trait should use the #[PostCondition] attribute instead of #[After] (or the @postCondition annotation instead of the @after annotation if you want/have to use annotations).

Since I was just working on a project using this package, I tried it and confirm changing @postCondition worked.

[mfn]

-> https://github.com/spatie/phpunit-snapshot-assertions/pull/168