Shouldn't it rather be failOnPassedOrSkippedAfterRetry?

[ivans083]

If a test is set to be skipped on it's body if certain condition meets, a retried test that is later skipped will fail the build even if failOnPassedAfterRetry = false. We could either fix and call it failOnPassedOrSkippedAfterRetry Or add failOnSkippedAfterRetry option in configuration

[buitcj]

I also would love this

[pshevche]

@ivans083 , @buitcj , if this is still relevant, could you please describe your use case better (maybe give us an example of such a volatile precondition)? From my point of view, failing the build in this case makes sense since there are failing tests that did not succeed. I'd also expect the condition to have a stable value and skip the test on every run.

For future reference, this could be used as a reproducer:

package org.example;

import org.junit.jupiter.api.Test;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

import static org.junit.jupiter.api.Assertions.fail;
import static org.junit.jupiter.api.Assumptions.assumeFalse;

class LibraryTest {
    private static final Path MARKER = Paths.get("build/flag");

    @Test
    void someLibraryMethodReturnsTrue() throws IOException {
        assumeFalse(Files.exists(MARKER));

        Files.createFile(MARKER);
        fail("test goes boom!");
    }
}

[adboyarshinov]

In my case, tests cases, that have known bugs are marked with special method knownBUG("BUG-123"), that set test to skipped and link issue BUG-123 to report. So in test report all skipped tests = known bugs. Problem occurs then fail accidentally happens before knownBug("BUG-123"), and in retry successfully reaches this method.

[pshevche]

@adboyarshinov , thank you for clarifying the use case. We've discussed it internally, and your proposal makes sense. Let us continue with the PR. Ping me on the PR if you need any help from my side.