search

awslabs / aws-sdk-swift

Apache License 2.0
399 stars 79 forks source link

Thread-leak in CRT engine when internet connection is cut midway #1534

Open sichanyoo opened 5 months ago

sichanyoo commented 5 months ago

Background

When using CRT HTTP client to get a 100MB file from a S3 bucket using getObject AND internet disconnects midway, it was observed that thread leak warning is thrown by the thread sanitizer. Note that if thread sanitizer is unchecked in Xcode, no indicator of the leak occurs.

Below is the log. Note that number of Body chunk received logs have been cut and replaced with <SKIPPED> as they clog up the log needlessly.

CLITool(77001,0x1fb5ebac0) malloc: nano zone abandoned due to inability to reserve vm space.
2024-05-30T16:29:44-0700 info S3Client : [Logging] Request: GET https:
Path: /100MiB.txt
Headers: Host: share-bucket-sichanyoo.s3.us-west-2.amazonaws.com,
X-Amz-Date: 20240530T232944Z,
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,
Authorization: AWS4-HMAC-SHA256 Credential=AKIA2NVENHBZH6NQSCUD/20240530/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=89901dd8f63a2c487d61f923197469240b1eb9585a877d5e1f883dee567521b4,
User-Agent: aws-sdk-swift/1.0 ua/2.0 api/s3#1.0 os/macos#14.4.1 lang/swift#5.10 cfg/retry-mode#legacy
Query: x-id=GetObject
2024-05-30T16:29:44-0700 info SerialExecutor : [Logging] Creating connection pool for share-bucket-sichanyoo.s3.us-west-2.amazonaws.com with max connections: 50
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Connection was acquired to: Optional("https://share-bucket-sichanyoo.s3.us-west-2.amazonaws.com/100MiB.txt?x-id=GetObject")
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Using HTTP/1.1 connection
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Main headers received
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Body chunk received
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Body chunk received
2024-05-30T16:29:45-0700 info S3Client : [Logging] User requested checksum validation, but the response headers did not contain any valid checksums
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Body chunk received
2024-05-30T16:29:45-0700 info CRTClientEngine : [Logging] Body chunk received

<SKIPPED>

2024-05-30T16:29:47-0700 info CRTClientEngine : [Logging] Body chunk received
2024-05-30T16:29:47-0700 info CRTClientEngine : [Logging] Body chunk received
2024-05-30T16:30:47-0700 info CRTClientEngine : [Logging] Request/response completed
2024-05-30T16:30:47-0700 info CRTClientEngine : [Logging] Response encountered an error: crtError(AwsCommonRuntimeKit.CRTError(code: 2073, message: "Http connection channel shut down due to failure to meet throughput minimum", name: "AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE"))

WARNING: ThreadSanitizer: thread leak (pid=77001)
Thread T6 (tid=10110039, finished) created by thread T1 at:
#0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x3062c)
#1 aws_thread_launch <null> (CLITool:arm64+0x1002c5f84)
#2 create_and_init_host_entry <null> (CLITool:arm64+0x10031eb00)
#3 default_resolve_host <null> (CLITool:arm64+0x10031cc9c)
#4 aws_host_resolver_resolve_host <null> (CLITool:arm64+0x10031b800)
#5 aws_client_bootstrap_new_socket_channel <null> (CLITool:arm64+0x100300970)
#6 aws_http_client_connect_internal <null> (CLITool:arm64+0x10035392c)
#7 aws_http_client_connect <null> (CLITool:arm64+0x1003549a4)
#8 s_aws_http_connection_manager_new_connection <null> (CLITool:arm64+0x10035bf94)
#9 s_aws_http_connection_manager_execute_transaction <null> (CLITool:arm64+0x1003585f4)
#10 aws_http_connection_manager_acquire_connection <null> (CLITool:arm64+0x100358bac)
#11 static AwsCommonRuntimeKit.HTTPClientConnectionManagerCallbackCore.acquireConnection(continuation: Swift.CheckedContinuation<AwsCommonRuntimeKit.HTTPClientConnection, Swift.Error>, connectionManager: AwsCommonRuntimeKit.HTTPClientConnectionManager) → () <null> (CLITool:arm64+0x100232b3c)
#12 closure #1 (Swift.CheckedContinuation<AwsCommonRuntimeKit.HTTPClientConnection, Swift.Error>) → () in AwsCommonRuntimeKit.HTTPClientConnectionManager.acquireConnection() async throws → AwsCommonRuntimeKit.HTTPClientConnection <null> (CLITool:arm64+0x10023235c)
#13 partial apply forwarder for closure #1 (Swift.CheckedContinuation<AwsCommonRuntimeKit.HTTPClientConnection, Swift.Error>) → () in AwsCommonRuntimeKit.HTTPClientConnectionManager.acquireConnection() async throws → AwsCommonRuntimeKit.HTTPClientConnection <null> (CLITool:arm64+0x1002323b4)
#14 merged closure #1 (Swift.UnsafeContinuation<A, Swift.Never>) → () in Swift.withCheckedContinuation<A>(function: Swift.String, _: (Swift.CheckedContinuation<A, Swift.Never>) → ()) async → A <null> (libswift_Concurrency.dylib:arm64e+0x63c4)

SUMMARY: ThreadSanitizer: thread leak (/Users/chanyoo/Library/Developer/Xcode/DerivedData/SDKWorkbench-fbjdenesfcxsoyawnpgfulljkaxs/Build/Products/Debug/CLITool:arm64+0x1002c5f84) in aws_thread_launch+0x340

ThreadSanitizer report breakpoint hit. Use 'thread info -s' to get extended information about the report.

ThreadSanitizer: reported 1 warnings
Program ended with exit code: 66
sichanyoo commented 4 months ago

Will pick up down the line if this issue resurfaces