Off-thread continuation resumes are slow on Linux

[oxy]

Summary:

Starting a continuation on one thread, and then firing the resume on another, when there is currently no other work being done, results in ~30 microseconds of overhead on Linux, as opposed to ~7 microseconds on macOS for the same benchmark, on the same hardware (Xcode 15 beta on macOS, Swift 5.8.1 on Linux).

Steps To Reproduce:

1. See attached benchmark (uploaded as .txt to make GitHub happy):
   • start continuation on one thread
   • write the continuation into a pipe
   • block on read in a detached task to resume
2. Run the same benchmark on macOS and Linux and observe significantly different results.

Results:

I expected that performance for the two platforms would be vaguely in-line with each other, but the continuation resume step is significantly slower on Linux - a profile verified that it was continuation resumes and not the relative performance of pipes on the two platforms.

Notes:

Some of the latency appears to be caused by the use of pthread semaphores for signaling on Linux in libdispatch.

Tracking as rdar://113640087.

[glessard]

The benchmark:

import Foundation

class AsyncPipe {
    let pipe: Pipe
    let task: Task<(), Error>
    typealias continuation = UnsafeContinuation<Int, Never>

    init() {
        self.pipe = Pipe()
        self.task = Task.detached { [pipe = self.pipe] in
            while (!Task.isCancelled) {
                let data = try pipe.fileHandleForReading.read(upToCount: 8)!
                let raw = data.reduce(0) { $0 << 8 + UInt64($1) }
                let cont = unsafeBitCast(raw, to: continuation.self)
                cont.resume(returning: Int.random(in: 0...4))
            }
        }
    }

    func nop() async -> Int {
        return await withUnsafeContinuation() { cont in
            let raw = unsafeBitCast(cont, to: UInt64.self)

            withUnsafeBytes(of: raw.bigEndian) { buf in
                try! self.pipe.fileHandleForWriting.write(contentsOf: buf)
            }
        }
    }

    deinit {
        self.task.cancel()
    }
}

func pipeBenchmark() async {
    let ring = AsyncPipe()
    var sum = 0
    for _ in 1...100000 {
        sum += await ring.nop()
    }
    print("Done! Random output:", sum)
}

await pipeBenchmark()