Cannot filter test names with commas using `swift test --filter`

[younata]

Hello!

I maintain the Quick testing framework. This issue comes as a result of a bug reported to Quick. Quick provides a DSL for organizing tests that, amongst other things, allows you to use any arbitrary string as the name of a test. For example, the following bit from Quick's DSL:

class SomeSpec: QuickSpec {
    override class func spec() {
        it("can include any character you want, including commas") {
            // whatever test you want it to be.
        }
    }
}

This code sample will generate a test, using xcodebuild's test filtering syntax, named "SomeSpec"/"can include any character you want, including commas" (double quotes added to make it easier to differentiate).

On Darwin platforms, this takes advantage of the fact that objc selectors can be any UTF-8 string. On non-Darwin platforms, this uses the allTests infrastructure to name tests. That is to say, this is not a bug in Quick's XCTest integration, you can easily reproduce this in pure XCTest (in SPM) with the following code:

class SomeTest: XCTestCase {
    static var allTests: [(String, (SomeTest) -> () throws -> Void)] {
        [("XCTest totally works with commas, right?", someTestMethod)]
    }

    func someTestMethod()
}

The bug is that, using swift test --filter, you can not select tests with commas in their names. That is, in the first example, swift test --filter "SomeSpec"/"can include any character you want, including commas or /s" will not find any tests to run. Similarly, with the second example, `swift test --filter "SomeTest/"XCTest totally can use /s or commas, right?" will not run any tests.

As @natinusala reported to us, this looks to be caused by XCTest's argument parser not considering that test names might contain commas. This causes XCTest to behave as if you wanted to filter tests named "SomeTest"/"XCTest totally works with commas" and " right?", which will almost always result in no tests found that match those filters (unless you have an XCTestCase subclass that shares the same name as the characters after the comma).

FWIW, this issue also exists when you use xcodebuild on Darwin, only the character that xcodebuild doesn't like filtering on is a slash.

[natinusala]

@younata I actually already made an issue back when I originally reported this: https://github.com/apple/swift-corelibs-xctest/issues/417

Take whatever you want from it (such as the project example to reproduce the bug) and I'll close mine as duplicate of yours.

[grynspan]

Duplicate of #417