HTTP/2 Async API - Githubissues

rnro commented 11 months ago

We previously developed an async API for NIO HTTP/2 which was guarded under SPI. Now that the swift-nio async API is released we can reintroduce this code promoted to SPI.

This change introduces:

AsyncStreamMultiplexer - an async variant of the HTTP/2 stream multiplexer which can be used to create outbound streams and provide access to an async sequence (NIOHTTP2AsyncSequence) of inbound streams
New pipeline configuration functions (e.g. configureAsyncHTTP2Pipeline) to support the new async mode

finagolfin commented 11 months ago

This pull broke the tests on Android and linux, as first surfaced by my daily Android CI yesterday:

Test CaSimultaneous accesses to 0x7c2f48fe5350, but modification requires exclusive access.
Previous access (a modification) started at swift-nio-http2PackageTests.xctest`<unavailable> + 12755447 (0x7c2f4df6e1f7).
Current access (a modification) started at:
0    libswiftCore.so                    0x00007c2f4a0c079e <unavailable> + 4872094

The reason it probably passed CI is because it appears to be some kind of race, as it is not always reproducible. I can fairly consistently reproduce on slower hardware though, where passing in --skip ConfiguringPipelineAsyncMultiplexerTests gets the remaining tests to pass again. I've been able to reproduce when building with Swift 5.9.1 on linux x86_64 and both Android AArch64 and x86_64 (this last platform shows the same error with the latest 5.10 and 5.11 snapshots also).

Lukasa commented 11 months ago

@FranzBusch can you take a look at this?

FranzBusch commented 11 months ago

@finagolfin I just ran the ConfiguringPipelineAsyncMultiplexerTests 10k times locally and wasn't able to reproduce this. Could you by any chance provide us a symbolicated backtrace when the test crashes? It also looks like some of your pipelines are not crashing in ConfiguringPipelineAsyncMultiplexerTests but in CompoundOutboundBufferTest.

finagolfin commented 11 months ago

Here's the full error output from a single-core linux x86_64 VPS, with this repo checked out to this commit 462ded7:

> ../swift-5.9.1-RELEASE-ubuntu20.04/usr/bin/swift test
--- snip passing tests ---
Test Case 'ConfiguringPipelineAsyncMultiplexerTests.testNIOAsyncConnectionStreamChannelPipelineCommunicates' started at 2023-10-27 08:32:35.706
Simultaneous accesses to 0x7f2210005280, but modification requires exclusive access.
Previous access (a modification) started at swift-nio-http2PackageTests.xctest`<unavailable> + 1306855 (0x563fa43310e7).
Current access (a modification) started at:
0    libswiftCore.so                    0x00007f2219e2edfc <unavailable> + 4218364
1    libswiftCore.so                    0x00007f2219e2ef00 swift_beginAccess + 66
2    swift-nio-http2PackageTests.xctest 0x0000563fa4330fae <unavailable> + 1306542
3    swift-nio-http2PackageTests.xctest 0x0000563fa4331e95 <unavailable> + 1310357
4    swift-nio-http2PackageTests.xctest 0x0000563fa43320fc <unavailable> + 1310972
5    swift-nio-http2PackageTests.xctest 0x0000563fa43384de <unavailable> + 1336542
6    swift-nio-http2PackageTests.xctest 0x0000563fa4338037 <unavailable> + 1335351
7    swift-nio-http2PackageTests.xctest 0x0000563fa43388dd <unavailable> + 1337565
8    libswiftCore.so                    0x00007f2219e30ff0 <unavailable> + 4227056
9    libswiftCore.so                    0x00007f2219e319ab <unavailable> + 4229547
10   swift-nio-http2PackageTests.xctest 0x0000563fa43310f8 <unavailable> + 1306872
11   swift-nio-http2PackageTests.xctest 0x0000563fa4331759 <unavailable> + 1308505
12   swift-nio-http2PackageTests.xctest 0x0000563fa440505a <unavailable> + 2175066
13   swift-nio-http2PackageTests.xctest 0x0000563fa4404e5d <unavailable> + 2174557
14   swift-nio-http2PackageTests.xctest 0x0000563fa4405209 <unavailable> + 2175497
15   swift-nio-http2PackageTests.xctest 0x0000563fa43f4321 <unavailable> + 2106145
16   swift-nio-http2PackageTests.xctest 0x0000563fa46f21af <unavailable> + 5243311
17   swift-nio-http2PackageTests.xctest 0x0000563fa44a465f <unavailable> + 2827871
18   swift-nio-http2PackageTests.xctest 0x0000563fa44a8854 <unavailable> + 2844756
19   swift-nio-http2PackageTests.xctest 0x0000563fa44a3b71 <unavailable> + 2825073
20   swift-nio-http2PackageTests.xctest 0x0000563fa44a4fbc <unavailable> + 2830268
21   swift-nio-http2PackageTests.xctest 0x0000563fa4433948 <unavailable> + 2365768
22   libdispatch.so                     0x00007f2219785187 <unavailable> + 147847
23   libdispatch.so                     0x00007f221978f8c8 <unavailable> + 190664
24   libdispatch.so                     0x00007f22197904a9 <unavailable> + 193705
25   libdispatch.so                     0x00007f22197972d6 <unavailable> + 221910
26   libpthread.so.0                    0x00007f2218cbc609 <unavailable> + 34313
27   libc.so.6                          0x00007f2218bd60f0 clone + 67
Fatal access conflict detected.
*** Program crashed: Aborted at 0x000003e9001818db ***
Thread 0 "swift-nio-http2":
0  0x00007f2218bc9a96 <unknown> in libc-2.31.so
Thread 1:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 2:
0  0x00007f2218bd646e <unknown> in libc-2.31.so
Thread 3:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 4:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 5:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 6 crashed:
0               0x00007f2218afa00b <unknown> in libc-2.31.so
 1 [ra]          0x00007f2219e2ea8b swift::fatalError(unsigned int, char const*, ...) + 122 in libswiftCore.so
 2 [ra]          0x00007f2219e2eea2 swift::runtime::AccessSet::insert(swift::runtime::Access*, void*, void*, swift::ExclusivityFlags) + 609 in libswiftCore.so
 3 [ra]          0x00007f2219e2ef42 swift_beginAccess + 65 in libswiftCore.so
 4 [ra]          0x0000563fa4330fae NIOAsyncChannelOutboundWriterHandler._didTerminate(error:) + 509 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/AsyncChannel/AsyncChannelOutboundWriterHandler.swift:110:19
 5 [ra]          0x0000563fa4331e95 NIOAsyncChannelOutboundWriterHandler.Delegate.didTerminate(error:) + 164 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/AsyncChannel/AsyncChannelOutboundWriterHandler.swift:208:30
 6 [ra] [thunk]  0x0000563fa43320fc protocol witness for NIOAsyncWriterSinkDelegate.didTerminate(error:) in conformance NIOAsyncChannelOutboundWriterHandler<A>.Delegate + 27 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/<compiler-generated>
 7 [ra]          0x0000563fa43384de NIOAsyncWriter.Storage.sinkFinish(error:) + 1149 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/AsyncSequences/NIOAsyncWriter.swift:622:26
 8 [ra]          0x0000563fa4338037 NIOAsyncWriter.Sink.InternalClass.deinit + 70 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/AsyncSequences/NIOAsyncWriter.swift:326:31
 9 [ra] [system] 0x0000563fa43388dd NIOAsyncWriter.Sink.InternalClass.__deallocating_deinit + 44 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/AsyncSequences/NIOAsyncWriter.swift
10 [ra]          0x00007f2219e30ff0 _swift_release_dealloc + 15 in libswiftCore.so
11 [ra]          0x00007f2219e319ab bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1> >::doDecrementSlow<(swift::PerformDeinit)1>(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int) + 122 in libswiftCore.so
12 [ra]          0x0000563fa43310f8 NIOAsyncChannelOutboundWriterHandler.handlerRemoved(context:) + 151 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/AsyncChannel/AsyncChannelOutboundWriterHandler.swift:136:19
13 [ra] [thunk]  0x0000563fa4331759 protocol witness for ChannelHandler.handlerRemoved(context:) in conformance NIOAsyncChannelOutboundWriterHandler<A> + 8 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/<compiler-generated>
14 [ra]          0x0000563fa440505a ChannelHandlerContext.invokeHandlerRemoved() + 217 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/ChannelPipeline.swift:1846:17
15 [ra]          0x0000563fa4404e5d ChannelPipeline.removeHandlerFromPipeline(context:promise:) + 732 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/ChannelPipeline.swift:575:17
16 [ra]          0x0000563fa4405209 ChannelPipeline.removeHandlers() + 408 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/ChannelPipeline.swift:601:17
17 [ra]          0x0000563fa43f4321 ChannelCore.removeHandlers(pipeline:) + 48 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOCore/Channel.swift:314:18
18 [ra]          0x0000563fa46f21af closure #2 in HTTP2StreamChannel.closedCleanly() + 110 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/Sources/NIOHTTP2/HTTP2StreamChannel.swift:648:18
19 [ra] [thunk]  0x0000563fa44a465f thunk for @escaping @callee_guaranteed () -> (@error @owned Error) + 14 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/<compiler-generated>
20 [ra] [thunk]
0x0000563fa44a8854 partial apply for thunk for @escaping @callee_guaranteed () -> (@error @owned Error) + 19 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/<compiler-generated>
21 [ra]          0x0000563fa44a3b71 closure #1 in NIOAsyncTestingEventLoop.insertTask<A>(taskID:deadline:promise:task:) + 128 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOEmbedded/AsyncTestingEventLoop.swift:133:37
22 [ra]          0x0000563fa44a4fbc closure #1 in NIOAsyncTestingEventLoop.execute(_:) + 2347 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/.build/checkouts/swift-nio/Sources/NIOEmbedded/AsyncTestingEventLoop.swift:197:30
23 [ra] [thunk]  0x0000563fa4433948 thunk for @escaping @callee_guaranteed () -> () + 39 in swift-nio-http2PackageTests.xctest at /home/foo/swift-nio-http2/<compiler-generated>
Thread 7:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 8:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 9:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Thread 10:
0  0x00007f2218cc6678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so
Registers:
rax 0x0000000000000000  0
rdx 0x00007f2207fff700  00 f7 ff 07 22 7f 00 00 a0 41 00 10 22 7f 00 00  ·÷ÿ·"··· A··"···
rcx 0x00007f2218afa00b  48 8b 84 24 08 01 00 00 64 48 33 04 25 28 00 00  H··$····dH3·%(··
rbx 0x0000000000000000  0
rsi 0x0000000000000002  2
rdi 0x00007f2207ffd6d0  07 ea fb 3b fe ff ff ff a9 04 79 19 22 7f 00 00  ·êû;þÿÿÿ©·y·"···
rbp 0x00007f2207ffd960  40 da ff 07 22 7f 00 00 8b ea e2 19 22 7f 00 00  @Úÿ·"····êâ·"···
rsp 0x00007f2207ffd6d0  07 ea fb 3b fe ff ff ff a9 04 79 19 22 7f 00 00  ·êû;þÿÿÿ©·y·"···
r8 0x0000000000000000  0
r9 0x00007f2207ffd6d0  07 ea fb 3b fe ff ff ff a9 04 79 19 22 7f 00 00  ·êû;þÿÿÿ©·y·"···
r10 0x0000000000000008  8
r11 0x0000000000000246  582
r12 0x00007f2219fb179b  46 61 74 61 6c 20 61 63 63 65 73 73 20 63 6f 6e  Fatal access con
r13 0x0000000000000021  33
r14 0x00007f21f0001990  46 61 74 61 6c 20 61 63 63 65 73 73 20 63 6f 6e  Fatal access con
r15 0x00007f2207ffda20  10 00 00 00 30 00 00 00 50 da ff 07 22 7f 00 00  ····0···PÚÿ·"···
rip 0x00007f2218afa00b  48 8b 84 24 08 01 00 00 64 48 33 04 25 28 00 00  H··$····dH3·%(··
rflags 0x0000000000000246  ZF PF
cs 0x0033  fs 0x0000  gs 0x0000
Images (20 omitted):
0x0000563fa41f2000–0x0000563fa5769378 <no build ID>                            swift-nio-http2PackageTests.xctest /home/foo/swift-nio-http2/.build/x86_64-unknown-linux-gnu/debug/swift-nio-http2PackageTests.xctest
0x00007f2218ab7000–0x00007f2218c50624 1878e6b475720c7c51969e69ab2d276fae6d1dee libc-2.31.so                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
0x00007f2218cb4000–0x00007f2218cca545 7b4536f41cdaa5888408e82d0836e33dcf436466 libpthread-2.31.so                 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
0x00007f2219a29000–0x00007f2219f6e888 <no build ID>                            libswiftCore.so                    /home/foo/swift-5.9.1-RELEASE-ubuntu20.04/usr/lib/swift/linux/libswiftCore.so
error: Exited with signal code 6

FranzBusch commented 10 months ago

Thanks for the backtrace @finagolfin. Just put up a PR to fix this: https://github.com/apple/swift-nio/pull/2580

FranzBusch commented 10 months ago

@finagolfin release with the fix is out: https://github.com/apple/swift-nio/releases/tag/2.61.0

apple / swift-nio-http2

HTTP/2 Async API #424