An error message is generated in certain cases when running `swift test --filter`.

[fibrechannelscsi]

Description

The code shown below generates an error message (error: malformed) when attempting to run swift test --filter. The argument passed to --filter does not appear to matter: invoking swift test --filter a is sufficient to reproduce the error.

Expected behavior

The unit test should complete successfully.

Actual behavior

The compilation process succeeds, but an error message saying error: malformed is generated, without the unit test actually running.

Steps to reproduce

This reproducer requires the following files; one of these is a Package.swift file. It requires one public imprort, swift-log.

The directory listing looks like this:

./Sources/main.swift
./Package.swift
./Tests/newPackage2Tests/a.swift

Note that ./Sources/main.swift can be empty, or set to its default contents after running a variation of swift package init.

Listing for Package.swift:

// swift-tools-version: 5.9
import PackageDescription
let package = Package(
    name: "newPackage",
    dependencies: [.package(url: "https://github.com/apple/swift-log.git", "1.4.0" ..< "2.0.0")],
    targets: [.testTarget(name: "newPackageTests", dependencies: [.product(name: "Logging", package: "swift-log")])])

Listing for ./Tests/newPackage2Tests/a.swift:

import Foundation; import XCTest; import Logging
let c = L(n: "")
class A
{
    public static func m() -> any X {var v = SystemRandomNumberGenerator(); let w = B.m(g: v.next())
        c.error("\(w)")
        return w
    }
}
final class Z: XCTestCase {var d = A.m(); func testV() throws {}}
public struct J: X {public init(e: UInt64) {}}
public protocol X {init(e: UInt64)}
public enum B {case m; public func callAsFunction(g: UInt64? = nil) -> any X {var a = SystemRandomNumberGenerator(); return self.y().init(e: a.next())}; public func y() -> any X.Type {return J.self}}
public func L(n: String) -> Logging.Logger {return Logging.Logger(label: n)}

Note that commenting out the line containing c.error causes the test to run successfully.

I have attached a log file (log0824.txt) that is based on the instructions recommended here: https://forums.swift.org/t/swift-test-sometimes-fails-with-error-malformed/66866/3 We create a script containing:

echo "$@" >> /tmp/log.txt
swiftc "$@"

and then run: SWIFT_EXEC=/path/to/script swift test log0824.txt

Swift Package Manager version/commit hash

2961cafb053b11d

Swift & OS version (output of swift --version ; uname -a)

Swift version 5.9-dev (LLVM f8c4926c34d7084, Swift 2961cafb053b11d)
Target: aarch64-unknown-linux-gnu
Linux 81ecb818bdf0 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

[fibrechannelscsi]

The resolved version of swift-log ended up being 1.5.3.

[finagolfin]

I can reproduce on Ubuntu 20.04 x86_64 with Swift 5.8.1.

[finagolfin]

Running the same command with --very-verbose (~the help is wrong, -vv doesn't produce the same output on linux~ I misread the help and typed -vv, but it says --vv, which does work) appears to turn up the bug. If I run without filtering, it simply runs the test runner /home/foo/goo/.build/x86_64-unknown-linux-gnu/debug/gooPackageTests.xctest and spits out this error, 2023-08-25T06:43:01+0000 error : [gooTests] J(), before the tests pass, no problem. That error is spit out whether in verbose mode or not, but it does nothing, ie it doesn't stop the tests.

However, if filtering, --very-verbose shows that it must first get a list of all tests with /home/foo/goo/.build/x86_64-unknown-linux-gnu/debug/gooPackageTests.xctest --dump-tests-json, so I ran that command manually and got:

2023-08-25T06:50:35+0000 error  : [gooTests] J()                  {"name":"All tests","tests":[{"tests":[{"tests":[{"name":"testV"}],"name":"gooTests.Z"}],"name":"debug.xctest"}]}

The problem appears to be that the same error is now dumped into the JSON file, which then causes the malformed error when the JSON is parsed?

[neonichu]

I guess that makes sense? If the test executable emits any output to stdout unconditionally, it messes with the idea that --dump-tests-json is assumed to only write the JSON to stdout.

Seems like a) SwiftPM should make more obvious what the issue is and b) the communication path for --dump-tests-json needs to be more robust (e.g. emit the output to a file explicitly, so it can't be mixed up with what is on stdout)

[neonichu]

How about this?

❯ $HOME/Library/Developer/Xcode/DerivedData/SwiftToolchain-hekdhwgdhbxkupdloruoivnoqmwu/Build/Products/Debug/swift-test --parallel --enable-experimental-test-output
Building for debugging...
Build complete! (0.04s)
error: invalid list test JSON structure, produced by /usr/libexec/swift/pm/swiftpm-xctest-helper /Users/neonacho/Desktop/simple2/.build/arm64-apple-macosx/debug/simple2PackageTests.xctest /var/folders/jc/zkdxhl4j0d35ydwqzd758l_w0000gn/T/TemporaryFile.7udd1a, underlying error: malformed(json: "garbage\n{\n  \"name\" : \"All Tests\",\n  \"tests\" : [\n    {\n      \"tests\" : [\n        {\n          \"name\" : \"simple2Tests.simple2Tests\",\n          \"tests\" : [\n            {\n              \"name\" : \"testExample\"\n            }\n          ]\n        }\n      ],\n      \"name\" : \"simple2PackageTests.xctest\"\n    }\n  ]\n}", underlyingError: Error Domain=NSCocoaErrorDomain Code=3840 "Invalid value around line 1, column 0." UserInfo={NSDebugDescription=Invalid value around line 1, column 0., NSJSONSerializationErrorIndex=0})

https://github.com/apple/swift-package-manager/pull/6860 https://github.com/apple/swift-tools-support-core/pull/429

[neonichu]

I'm probably not going to personally pursue improvements to --dump-tests-json for now, I think improvements to the diagnostics should at least allow people to work around this. The issue should stay open, though, of course.

[neonichu]

Merged the diagnostics improvements now, thanks @fibrechannelscsi and @finagolfin for getting to the bottom of it!

[mapedd]

Hey, is there any way to work around this limitation?. Trying to run swift test with --filter option in Docker (swift-5.9.2-jammy) but getting : error: malformed

[finagolfin]

Don't have your tests dump output regardless of scenario. If that is too difficult, first run swift test -l to dump a list of all tests, then manually filter out tests you want and run those instead.

[mapedd]

My problem is I have single package that has :

1. unit tests
2. integration tests (need real server) I want to run just unit tests. Seems that I can simply use --skip option and might solve my problem

[mapedd]

Seems it fails too :( I'm trying to use swift test --test-product but to no avail Looks like only solution is to create a separate Package.swift that defines only tests I want to run. Sigh...

[finagolfin]

That may not work either, if those tests are dumping output too. This is a SPM bug, where tests that indiscriminately dump output affect how the tests can be run.

@bnbarham, maybe a dev can be assigned to finally fix this, since it is affecting more people now, perhaps by making "the communication path for --dump-tests-json needs to be more robust (e.g. emit the output to a file explicitly, so it can't be mixed up with what is on stdout)," as Boris suggested above last year.