Crash on 1.9.0: Precondition Failure in HTTP2IdleHandler

[qusc]

I've experienced this crash multiple times on my server:

Received signal 4. Backtrace:
0x55b7c15c82c2, Backtrace.(printBacktrace in _B82A8C0ED7C904841114FDF244F9E58E)(signal: Swift.Int32) -> () at /swift-backtrace/Sources/Backtrace/Backtrace.swift:66
0x7faebbae37df
0x55b7c143256f, Swift runtime failure: precondition failure at /async-http-client/Sources/AsyncHTTPClient/ConnectionPool/HTTP2/HTTP2IdleHandler.swift:0
0x55b7c143256f, AsyncHTTPClient.HTTP2IdleHandler.StateMachine.goAwayReceived() -> AsyncHTTPClient.HTTP2IdleHandler<A>.StateMachine.Action at /async-http-client/Sources/AsyncHTTPClient/ConnectionPool/HTTP2/HTTP2IdleHandler.swift:197
0x55b7c143256f, AsyncHTTPClient.HTTP2IdleHandler.channelRead(context: NIOCore.ChannelHandlerContext, data: NIOCore.NIOAny) -> () at /async-http-client/Sources/AsyncHTTPClient/ConnectionPool/HTTP2/HTTP2IdleHandler.swift:66
0x55b7c1aa2772, NIOCore.ChannelHandlerContext.(invokeChannelRead in _F5AC316541457BD146E3694279514AA3)(NIOCore.NIOAny) -> () at /swift-nio/Sources/NIOCore/ChannelPipeline.swift:1688
0x55b7c1aa026f, NIOCore.ChannelHandlerContext.fireChannelRead(NIOCore.NIOAny) -> () at /swift-nio/Sources/NIOCore/ChannelPipeline.swift:1501
0x55b7c1b8bbb0, NIOHTTP2.NIOHTTP2Handler.(processFrame in _57CB3A28E3B2E42C5D0799905DE90E42)(_: NIOHTTP2.HTTP2Frame, flowControlledLength: Swift.Int, context: NIOCore.ChannelHandlerContext) -> NIOHTTP2.NIOHTTP2Handler.FrameProcessResult at /swift-nio-http2/Sources/NIOHTTP2/HTTP2ChannelHandler.swift:304
0x55b7c1b88ece, NIOHTTP2.NIOHTTP2Handler.(frameDecodeLoop in _57CB3A28E3B2E42C5D0799905DE90E42)(context: NIOCore.ChannelHandlerContext) -> () at /swift-nio-http2/Sources/NIOHTTP2/HTTP2ChannelHandler.swift:199
0x55b7c1b88ece, NIOHTTP2.NIOHTTP2Handler.channelRead(context: NIOCore.ChannelHandlerContext, data: NIOCore.NIOAny) -> () at /swift-nio-http2/Sources/NIOHTTP2/HTTP2ChannelHandler.swift:163
0x55b7c1aa2772, NIOCore.ChannelHandlerContext.(invokeChannelRead in _F5AC316541457BD146E3694279514AA3)(NIOCore.NIOAny) -> () at /swift-nio/Sources/NIOCore/ChannelPipeline.swift:1688
0x55b7c1aa026f, NIOCore.ChannelHandlerContext.fireChannelRead(NIOCore.NIOAny) -> () at /swift-nio/Sources/NIOCore/ChannelPipeline.swift:1501
0x55b7c1c474e4, NIOSSL.NIOSSLHandler.(doFlushReadData in _4C55B9A85907C0CB3F4E7FBD2C1C5493)(context: NIOCore.ChannelHandlerContext, receiveBuffer: NIOCore.ByteBuffer, readOnEmptyBuffer: Swift.Bool) -> () at /swift-nio-ssl/Sources/NIOSSL/NIOSSLHandler.swift:436
0x55b7c1c45f1a, NIOSSL.NIOSSLHandler.channelReadComplete(context: NIOCore.ChannelHandlerContext) -> () at /swift-nio-ssl/Sources/NIOSSL/NIOSSLHandler.swift:165
0x55b7c1aa27ca, NIOCore.ChannelHandlerContext.(invokeChannelReadComplete in _F5AC316541457BD146E3694279514AA3)() -> () at /swift-nio/Sources/NIOCore/ChannelPipeline.swift:1698
0x55b7c1aa3462
0x55b7c1c1869a
0x55b7c1c1f7c7, generic specialization <NIOPosix.Socket> of NIOPosix.BaseSocketChannel.readEOF0() -> () at .build/checkouts/swift-nio/Sources/NIOPosix/BaseSocketChannel.swift:988
0x55b7c1c1f7c7, generic specialization <NIOPosix.Socket> of NIOPosix.BaseSocketChannel.readEOF() -> () at .build/checkouts/swift-nio/Sources/NIOPosix/BaseSocketChannel.swift:974
0x55b7c1c1f7c7, generic specialization <NIOPosix.Socket> of protocol witness for NIOPosix.SelectableChannel.readEOF() -> () in conformance NIOPosix.BaseSocketChannel<A> : NIOPosix.SelectableChannel in NIOPosix at /root/<compiler-generated>:965
0x55b7c1c1f7c7, function signature specialization <Arg[2] = Dead> of generic specialization <NIOPosix.SocketChannel> of NIOPosix.SelectableEventLoop.handleEvent<A where A: NIOPosix.SelectableChannel>(_: NIOPosix.SelectorEventSet, channel: A) -> () at /swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:392
0x55b7c1c1f242, reabstraction thunk helper from @callee_guaranteed (@guaranteed NIOPosix.SelectorEvent<NIOPosix.NIORegistration>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed NIOPosix.SelectorEvent<NIOPosix.NIORegistration>) -> (@error @owned Swift.Error) at /root/<compiler-generated>:0
0x55b7c1c1f242, partial apply forwarder for reabstraction thunk helper from @callee_guaranteed (@guaranteed NIOPosix.SelectorEvent<NIOPosix.NIORegistration>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed NIOPosix.SelectorEvent<NIOPosix.NIORegistration>) -> (@error @owned Swift.Error) at /root/<compiler-generated>:0
0x55b7c1c1b37c, generic specialization <NIOPosix.NIORegistration> of NIOPosix.Selector.whenReady0(strategy: NIOPosix.SelectorStrategy, onLoopBegin: () -> (), _: (NIOPosix.SelectorEvent<A>) throws -> ()) throws -> () at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorEpoll.swift:252
0x55b7c1c18abf, generic specialization <NIOPosix.NIORegistration> of NIOPosix.Selector.whenReady(strategy: NIOPosix.SelectorStrategy, onLoopBegin: () -> (), _: (NIOPosix.SelectorEvent<A>) throws -> ()) throws -> () at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorGeneric.swift:286
0x55b7c1c18abf, closure #2 () throws -> () in NIOPosix.SelectableEventLoop.run() throws -> () at /swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:461
0x55b7c1c18abf, reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error) at /root/<compiler-generated>:0
0x55b7c1c18abf, generic specialization <()> of NIOPosix.withAutoReleasePool<A>(() throws -> A) throws -> A at /swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:28
0x55b7c1c18abf, NIOPosix.SelectableEventLoop.run() throws -> () at /swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:460
0x55b7c1c04084, static NIOPosix.MultiThreadedEventLoopGroup.(runTheLoop in _C2B1528F4FBA68A3DBFA89DBAEBE9D4D)(thread: NIOPosix.NIOThread, parentGroup: Swift.Optional<NIOPosix.MultiThreadedEventLoopGroup>, canEventLoopBeShutdownIndividually: Swift.Bool, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>, initializer: (NIOPosix.NIOThread) -> (), _: (NIOPosix.SelectableEventLoop) -> ()) -> () at /swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:89
0x55b7c1c04084, closure #1 (NIOPosix.NIOThread) -> () in static NIOPosix.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _C2B1528F4FBA68A3DBFA89DBAEBE9D4D)(name: Swift.String, parentGroup: NIOPosix.MultiThreadedEventLoopGroup, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>, initializer: (NIOPosix.NIOThread) -> ()) -> NIOPosix.SelectableEventLoop at /swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:110
0x55b7c1c06bce, partial apply forwarder for closure #1 (NIOPosix.NIOThread) -> () in static NIOPosix.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _C2B1528F4FBA68A3DBFA89DBAEBE9D4D)(name: Swift.String, parentGroup: NIOPosix.MultiThreadedEventLoopGroup, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>, initializer: (NIOPosix.NIOThread) -> ()) -> NIOPosix.SelectableEventLoop at /root/<compiler-generated>:0
0x55b7c1c06ec9, reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIOPosix.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIOPosix.NIOThread) -> (@out ()) at /root/<compiler-generated>:0
0x55b7c1c06bed, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIOPosix.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIOPosix.NIOThread) -> (@out ()) at /root/<compiler-generated>:0
0x55b7c1c335e5, closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIOPosix.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIOPosix.Box<(body: (NIOPosix.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at /swift-nio/Sources/NIOPosix/ThreadPosix.swift:105
0x7faebbad940a
0x7faeb99a440e
0xffffffffffffffff

[Lukasa]

Thanks for this! I think this is a straightforward bug, but we're a little short-staffed at the moment so we probably won't get a look at this until next week.

[qusc]

Great, thanks!

[rausnitz]

We are also encountering this issue. @qusc have you been able to figure out how this crash gets triggered? If I can pinpoint that I might just turn off the feature that's leading to the crash.

[rausnitz]

On closer inspection we've actually got a different crash than this. Issue filed here: https://github.com/swift-server/async-http-client/issues/576.