Using recording an issue inside of a detached task causes a crash.

[younata]

Description

This is something I learned while researching potential ways to address #475. Which, at the time I wrote that issue, I thought would have just resulted in the failure being silently dropped.

The current way of determining what the current reporting tests natively in Swift-Testing is broken when you run an expect inside of a detached task. Technically, you get a failing test result, but it's because the test runner crashed. I haven't yet spelunked enough through the codebase to figure out where that crash is.

This is because Test.current is stored as a task-local value. Which means that when you make an expect inside of a detached task, Swift Testing has lost that context, which means that Test.current is incorrectly (from the semantics of the current running test) as nil.

I don't know the concurrency system well enough to be able to suggest a good solution. The issue is that, because Swift Testing supports task-based parallelization, the value of Test.current strongly depends on the task it's running on. Which makes having a global Test.current like what's in swift-corelibs-xctest the wrong solution.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

This test should pass. The test runner crashes with a non-zero error instead.

import Testing

@Test func CheckingExpectWithDetachedTasks() async {
    await Task.detached {
        #expect(Bool(false))
    }.value
}

swift-testing version/commit hash

Testing Library Version: 75.7 (arm64e-apple-macos13.0)

Swift & OS version (output of swift --version ; uname -a)

No response

[stmontgomery]

The crash here is a known issue in Xcode 16 Beta, tracked by rdar://122190668. To resolve it, some work will be needed on the Xcode side, but it's also likely some additional action will be needed here in the swift-testing codebase, so I will leave this issue open to track that side of things.

[grynspan]

(To clarify: the crash is specific to Xcode 16.)

[younata]

Having not tried Swift Testing on Xcode 15, is the behavior that it silently drops the issue?

[stmontgomery]

(To clarify: the crash is specific to Xcode 16.)

Right. But that said, other tools (such as VS Code or swift test CLI) may not be handling this situation very clearly in the reporting either, despite not crashing. Because of parallel execution, it's not always obvious which test a particular Issue is associated with. One idea I've had is to keep track of all the currently-running tests and include them as "potentially associated" tests in the reporting, so that at least you can narrow the scope down when troubleshooting.

[grynspan]

To clarify more: the crash is specific to Xcode 16, which is the first version of Xcode to include support for Swift Testing. I meant that the crash is a bug in Xcode specifically, and not workflows that involve e.g. Linux or a package dependency on Swift Testing rather than Xcode and the copy of Swift Testing embedded in it.

I hope that makes more sense!

[younata]

To clarify more: the crash is specific to Xcode 16, which is the first version of Xcode to include support for Swift Testing. I meant that the crash is a bug in Xcode specifically, and workflows that involve e.g. Linux or a package dependency on Swift Testing rather than Xcode and the copy of Swift Testing embedded in it.

I hope that makes more sense!

Oh! I gotcha! Thanks!

[tonyarnold]

Folks, did a fix for this make it into Xcode 16.x yet? I'm seeing the same thing both locally and on CI:

XCTest/HarnessEventHandler.swift:282: Fatal error: Internal inconsistency: No test reporter for test case argumentIDs: Optional([Testing.Test.Case.Argument.ID(bytes: [34, 67, 52, 51, 56, 68, 66, 52, 65, 45, 57, 54, 70, 67, 45, 52, 70, 56, 66, 45, 66, 69, 48, 52, 45, 57, 53, 51, 56, 52, 69, 55, 53, 49, 49, 65, 57, 34])]) in test FormAssistantTests.PrefixedUniqueIdentifiableTests/testPrefixIsCorrectlyPrepended(to:)/PrefixedUniqueIdentifiableTests.swift:24:6

[soapyfrog]

Interestingly, one project (SPM) I have was generally fine with Xcode 16.0 on macOS 15.0 but this is happening inconsistently (about 50% of the time) since 16.1/15.1 respectively.

I see there's an update in Xcode command-line tools coming down the pipe now (Oct 29 23:27 GMT), so will see if that makes a difference.

[grynspan]

Xcode CL Tools don't include the component that causes the crash (as far as I'm aware, anyway.)

[soapyfrog]

Xcode CL Tools don't include the component that causes the crash (as far as I'm aware, anyway.)

Yes, it made no difference. Didn't think it would; just wishful thinking. This is a really frustrating bug as since 16.1 on 15.1 I'm having to run all my tests with swift test instead of ⌘U in Xcode.

[grynspan]

I'm sorry it's not going well! Apple is tracking this issue, but since Xcode is not part of the open source Swift project, there's not a lot we can say publicly about it.

[soapyfrog]

No worries. I appreciate the correspondence 👍