Steps are reported as passing although the step definition's closure is XCTFail

[jfahrenkrug]

Is your feature request related to a problem? Please describe. I have multiple step definitions, and currently they all just execute XCTFail. However, two out of three of my feature steps are reported as successful tests. I would expect all tests to be reported as failures in Xcode. Also, continueTestingAfterFailure is true.

Describe the solution you'd like I would expect all steps that have a matching step definition to be reported as failures if the closure of those step definitions is running XCTFail.

Describe alternatives you've considered I've removed XCTFail from the first step definition. When I do that, the first step passes and then the 2nd step fails and the 3rd step is reported as a success again. Also, continueTestingAfterFailure doesn't seem to make a difference.

Additional context

My step definitions:

import XCTest
import CucumberSwift

extension Cucumber: StepImplementation {
    public var bundle: Bundle {
        Bundle.module
    }

    public var continueTestingAfterFailure: Bool {
        true
    }

    public func setupSteps() {

        Given(#/^a key \"([^\"]*)\"$/#) { matches, _ in
            XCTFail("not implemented")
        }

        When(#/^I validate the signature of [\"]?([^\"]*)[\"]?$/#) { matches, _ in
            XCTFail("not implemented")
        }

        Then(#/^the validation result should be [\"]?([^\"]*)[\"]?$/#) { matches, _ in
            XCTFail("not implemented")
        }
    }
}

The feature:

Feature: Validation of signed files

   Scenario: File smaller than 64kb
      Given a key "somekey.pub"
      When I validate the signature of "testfile.zip"
      Then the validation result should be "SUCCESS"

The Xcode output:

[Tyler-Keith-Thompson]

Thanks to a break in Xcode 14.something the way you define the test bundle had to change. Can you try:

public var bundle: Bundle {
    class TestExplorer: CucumberTest { } // !! Make sure to inherit from CucumberTest
    return Bundle(for: TestExplorer.self) // !! Important, this is what allows Cucumber Tests to be discovered.
}

I'm hoping that resolves your issue and the weird behavior can be explained by Xcode weirdness.

[jfahrenkrug]

Hi @Tyler-Keith-Thompson,

Thanks so much for your reply. I'm using CucumberSwift inside of a Swift Package. In other words, I'm using it not in an xcodeproj, but to test a swift package I'm writing. I had to use Bundle.module to get the bundle path right to discover the features. That part works. But the "false positives" are the problem. Thanks again for your help and your great library.

[Tyler-Keith-Thompson]

Sure, I'd expect the feature files to be found but inheriting from CucumberTest actually sets up some of the XCTestObservation stuff.

So given you've got the use-case of only doing this as part of a Package.swift how about this. Try adding an XCTestCase that will reasonably execute, like this:

final class TestExplorerForSPM: CucumberTest {
    func testDiscoveryOfASingleTestToTriggerXCTestCorrectly() {
        XCTAssert(true)
    }
}

In the meantime I'll try to repro your setup, hadn't ever tried using this without an xcodeproj before but it's totally a use-case we should support.

[jfahrenkrug]

Thank you, I've added that XCTestCase, but it doesn't make a difference. It seems like if one step inside of a scenario fails, the remaining steps in that scenario are reported as successfully passing, although their matching step definition closures are not executed (I set breakpoints to verify that).

[Tyler-Keith-Thompson]

Just wanted to circle back, I haven't forgotten about this, I've just been swamped. I've got reason to suspect this is Xcode being eccentric. Curious if 14.3 has any different behavior.

[NFulkerson]

Running into the same issue here, it almost seems as if the matchers don't work correctly at all in a standalone Swift package. If I use Bundle.module the feature file is discovered properly but tests continue to claim no steps are implemented, and all tests pass.