search

apple / swift-nio

Event-driven network application framework for high performance protocol servers & clients, non-blocking.
https://swiftpackageindex.com/apple/swift-nio/documentation
Apache License 2.0
7.98k stars 650 forks source link

crash： free(): corrupted unsorted chunks #2617

Closed shenfu1991 closed 10 months ago

shenfu1991 commented 10 months ago

Expected behavior

[what you expected to happen] Stable running

Actual behavior

crashed

Steps to reproduce

My Vapor program crashed after receiving several network request errors.

If possible, minimal yet complete reproducer code (or URL to code)

see log

SwiftNIO version/commit hash

revision : "702cd7c56d5d44eeba73fdf83918339b26dc855c", version" : "2.62.0"

[the SwiftNIO tag/commit hash]

System & version information

Please provide at the very least your operating system and Swift version information.

OS: ubuntu20.04 swift version: 5.9.2

Ideally, run

scripts/nio-diagnose -o nio-diagnose.md PID_OF_YOUR_NIO_PROGRAM

and attach (or paste) the resulting file nio-diagnose.md into this bug report or send it to the SwiftNIO maintainers privately.

logs:

2023-12-30 00:00:45.314 cty_3[244969:68ff9700] 2023-12-30 00:00:45    loadFeatureBalance() 币安接口小黑屋...Received error status: 400 Bad Request    

2023-12-30 00:00:50.167 cty_3[244969:68ff9700] 2023-12-30 00:00:50    loadFeatureBalance() 币安接口小黑屋...Response error: HTTPClientError.getConnectionFromPoolTimeout,desc: The operation could not be completed. (AsyncHTTPClient.HTTPClientError error 1.)    

2023-12-30 00:00:51.678 cty_3[244969:68ff9700] 2023-12-30 00:00:51    loadFeatureBalance() 币安接口小黑屋...Received error status: 400 Bad Request    

free(): corrupted unsorted chunks

*** Signal 6: Backtracing from 0x7f387c23b00b... done ***

*** Program crashed: Aborted at 0x000000000003bce9 ***

Thread 0 "cty_3":

0  0x00007f387c23b48a  in libc-2.31.so

Thread 1 "NIO-ELT-0-#0":

0                0x00007f387c31768e  in libc-2.31.so
1 [ra] [system]  0x0000559f5a48ddb1 specialized Selector.whenReady0(strategy:onLoopBegin:_:) + 624 in cty_3 at /root/cty_3/x-bot/
2 [ra] [inlined] 0x0000559f5a48a85a specialized Selector.whenReady(strategy:onLoopBegin:_:) in cty_3 at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorGeneric.swift:288:18
3 [ra]           0x0000559f5a48a85a SelectableEventLoop.run() + 521 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:470:36
4 [ra] [inlined] 0x0000559f5a468d4f static MultiThreadedEventLoopGroup.runTheLoop(thread:parentGroup:canEventLoopBeShutdownIndividually:selectorFactory:initializer:_:) in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:93:22
5 [ra]           0x0000559f5a468d4f closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 334 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:114:41
6 [ra] [thunk]   0x0000559f5a46c433 partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 50 in cty_3 at /root/cty_3/x-bot/
7 [ra] [thunk]   0x0000559f5a46e95f thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () + 14 in cty_3 at /root/cty_3/x-bot/
8 [ra]           0x0000559f5a4a68fd closure #1 in static ThreadOpsPosix.run(handle:args:detachThread:) + 380 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/ThreadPosix.swift:112:13

Thread 2 "NIO-ELT-0-#1":

0                0x00007f387c31768e  in libc-2.31.so
1 [ra] [system]  0x0000559f5a48ddb1 specialized Selector.whenReady0(strategy:onLoopBegin:_:) + 624 in cty_3 at /root/cty_3/x-bot/
2 [ra] [inlined] 0x0000559f5a48a85a specialized Selector.whenReady(strategy:onLoopBegin:_:) in cty_3 at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorGeneric.swift:288:18
3 [ra]           0x0000559f5a48a85a SelectableEventLoop.run() + 521 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:470:36
4 [ra] [inlined] 0x0000559f5a468d4f static MultiThreadedEventLoopGroup.runTheLoop(thread:parentGroup:canEventLoopBeShutdownIndividually:selectorFactory:initializer:_:) in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:93:22
5 [ra]           0x0000559f5a468d4f closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 334 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:114:41
6 [ra] [thunk]   0x0000559f5a46c433 partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 50 in cty_3 at /root/cty_3/x-bot/
7 [ra] [thunk]   0x0000559f5a46e95f thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () + 14 in cty_3 at /root/cty_3/x-bot/
8 [ra]           0x0000559f5a4a68fd closure #1 in static ThreadOpsPosix.run(handle:args:detachThread:) + 380 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/ThreadPosix.swift:112:13

Thread 3 "NIO-ELT-0-#2":

0                0x00007f387c31768e  in libc-2.31.so
1 [ra] [system]  0x0000559f5a48ddb1 specialized Selector.whenReady0(strategy:onLoopBegin:_:) + 624 in cty_3 at /root/cty_3/x-bot/
2 [ra] [inlined] 0x0000559f5a48a85a specialized Selector.whenReady(strategy:onLoopBegin:_:) in cty_3 at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorGeneric.swift:288:18
3 [ra]           0x0000559f5a48a85a SelectableEventLoop.run() + 521 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:470:36
4 [ra] [inlined] 0x0000559f5a468d4f static MultiThreadedEventLoopGroup.runTheLoop(thread:parentGroup:canEventLoopBeShutdownIndividually:selectorFactory:initializer:_:) in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:93:22
5 [ra]           0x0000559f5a468d4f closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 334 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:114:41
6 [ra] [thunk]   0x0000559f5a46c433 partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 50 in cty_3 at /root/cty_3/x-bot/
7 [ra] [thunk]   0x0000559f5a46e95f thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () + 14 in cty_3 at /root/cty_3/x-bot/
8 [ra]           0x0000559f5a4a68fd closure #1 in static ThreadOpsPosix.run(handle:args:detachThread:) + 380 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/ThreadPosix.swift:112:13

Thread 4 "NIO-ELT-0-#3":

0                0x00007f387c31768e  in libc-2.31.so
1 [ra] [system]  0x0000559f5a48ddb1 specialized Selector.whenReady0(strategy:onLoopBegin:_:) + 624 in cty_3 at /root/cty_3/x-bot/
2 [ra] [inlined] 0x0000559f5a48a85a specialized Selector.whenReady(strategy:onLoopBegin:_:) in cty_3 at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorGeneric.swift:288:18
3 [ra]           0x0000559f5a48a85a SelectableEventLoop.run() + 521 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:470:36
4 [ra] [inlined] 0x0000559f5a468d4f static MultiThreadedEventLoopGroup.runTheLoop(thread:parentGroup:canEventLoopBeShutdownIndividually:selectorFactory:initializer:_:) in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:93:22
5 [ra]           0x0000559f5a468d4f closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 334 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:114:41
6 [ra] [thunk]   0x0000559f5a46c433 partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) + 50 in cty_3 at /root/cty_3/x-bot/
7 [ra] [thunk]   0x0000559f5a46e95f thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () + 14 in cty_3 at /root/cty_3/x-bot/
8 [ra]           0x0000559f5a4a68fd closure #1 in static ThreadOpsPosix.run(handle:args:detachThread:) + 380 in cty_3 at /root/cty_3/x-bot/.build/checkouts/swift-nio/Sources/NIOPosix/ThreadPosix.swift:112:13

Thread 5 "TP-#0":

0  0x00007f387c622454 do_futex_wait.constprop.0 + 52 in libpthread-2.31.so

Thread 6 "TP-#1":

0  0x00007f387c622454 do_futex_wait.constprop.0 + 52 in libpthread-2.31.so

Thread 7 "TP-#2":

0  0x00007f387c622454 do_futex_wait.constprop.0 + 52 in libpthread-2.31.so

Thread 8 "TP-#3":

0  0x00007f387c622454 do_futex_wait.constprop.0 + 52 in libpthread-2.31.so

Thread 9:

0  0x00007f387c622678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so

Thread 10:

0  0x00007f387c31768e  in libc-2.31.so

Thread 11:

0  0x00007f387c622678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so

Thread 12:

0  0x00007f387c622678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so

Thread 13 crashed:

0      0x00007f387c23b00b  in libc-2.31.so
1 [ra] 0x00007f387c28d2fc  in libc-2.31.so

Thread 14:

0  0x00007f387c622678 do_futex_wait.constprop.0 + 72 in libpthread-2.31.so

Registers:

rax 0x0000000000000000  0
rdx 0x00007f384effd700  00 d7 ff 4e 38 7f 00 00 70 0b 00 3c 38 7f 00 00  ·×ÿN8···p··<8···
rcx 0x00007f387c23b00b  48 8b 84 24 08 01 00 00 64 48 33 04 25 28 00 00  H··$····dH3·%(··
rbx 0x0000000000000000  0
rsi 0x0000000000000002  2
rdi 0x00007f384effc730  07 ea fb 3b fe ff ff ff 00 23 fc 15 8b 94 c6 97  ·êû;þÿÿÿ·#ü···Æ·
rbp 0x00007f384effca80  20 00 00 54 38 7f 00 00 fc d2 28 7c 38 7f 00 00   ··T8···üÒ(|8···
rsp 0x00007f384effc730  07 ea fb 3b fe ff ff ff 00 23 fc 15 8b 94 c6 97  ·êû;þÿÿÿ·#ü···Æ·
 r8 0x0000000000000000  0
 r9 0x00007f384effc730  07 ea fb 3b fe ff ff ff 00 23 fc 15 8b 94 c6 97  ·êû;þÿÿÿ·#ü···Æ·
r10 0x0000000000000008  8
r11 0x0000000000000246  582
r12 0x00007f384effc9a0  10 cb ff 4e 38 7f 00 00 00 57 4e 7d 38 7f 00 00  ·ËÿN8····WN}8···
r13 0x0000000000000010  16
r14 0x00007f387c783000  00 10 00 00 66 72 65 65 28 29 3a 20 63 6f 72 72  ····free(): corr
r15 0x0000000000000001  1
rip 0x00007f387c23b00b  48 8b 84 24 08 01 00 00 64 48 33 04 25 28 00 00  H··$····dH3·%(··

rflags 0x0000000000000246  ZF PF

cs 0x0033  fs 0x0000  gs 0x0000

Images (23 omitted):

0x0000559f59bf0000–0x0000559f5aa436d8                             cty_3              /root/cty_3/x-bot/.build/x86_64-unknown-linux-gnu/release/cty_3
0x00007f387c1f8000–0x00007f387c391864 eebe5d5f4b608b8a53ec446b63981bba373ca0ca libc-2.31.so       /usr/lib/x86_64-linux-gnu/libc-2.31.so
0x00007f387c610000–0x00007f387c626545 0c044ba611aeeeaebb8374e660061f341ebc0bac libpthread-2.31.so /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

Backtrace took 22.11s
Lukasa commented 10 months ago

The report here isn’t very detailed but it looks like this is a threading bug in your code. Does this crash reproduce on macOS?

Lukasa commented 10 months ago

Actually this appears to be a dupe of swift-server/async-http-client#724. You have more information there, so let’s use that report.