search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
2.91k stars 245 forks source link

Disable StalledStreamProtection for Lambda #1166

Closed wafflespeanut closed 1 week ago

wafflespeanut commented 2 weeks ago

Describe the bug

I'm facing ThroughputBelowMinimum whenever I invoke lambda. Pretty much looks like #1141, which has been marked as resolved, but when I use disable stalled stream protection, I'm able to get past it and actually get a response.

Expected Behavior

Succeeding Lambda function invocation whenever I want.

Current Behavior

Significant proportion of failures over successful requests

Reproduction Steps

Continuous lambda invocation.

Possible Solution

Disable StalledStreamProtection in client config.

Additional Information/Context

No response

Version

cargo tree | grep aws-
├── aws-config v1.5.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.7.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.2.0
│   │   ├── aws-smithy-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.2.0 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   │   │   ├── aws-smithy-types v1.2.0 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-types v1.3.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   │   ├── aws-smithy-types v1.2.0 (*)
│   ├── aws-sdk-sso v1.28.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-smithy-runtime v1.6.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-types v1.3.1 (*)
│   ├── aws-sdk-ssooidc v1.29.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.6.0 (*)
│   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-types v1.3.1 (*)
│   ├── aws-sdk-sts v1.28.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-smithy-runtime v1.6.0 (*)
│   │   ├── aws-smithy-runtime-api v1.7.0 (*)
│   │   ├── aws-smithy-types v1.2.0 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.3.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.6.0 (*)
│   ├── aws-smithy-runtime-api v1.7.0 (*)
│   ├── aws-smithy-types v1.2.0 (*)
│   ├── aws-types v1.3.1 (*)
├── aws-credential-types v1.2.0 (*)
├── aws-sdk-lambda v1.31.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.6.0 (*)
│   ├── aws-smithy-runtime-api v1.7.0 (*)
│   ├── aws-smithy-types v1.2.0 (*)
│   ├── aws-types v1.3.1 (*)
├── aws-types v1.3.1 (*)
│   ├── aws-sdk-lambda v1.31.0 (*)

Environment details (OS name and version, etc.)

MacOS 14.5 Sonoma (Silicon chip)

Logs

2024-06-13 12:30:59.223823  [DEBUG] current throughput: 0 B/s is below minimum: 1 B/s
2024-06-13 12:30:59.223826  [DEBUG] current throughput: 0 B/s is below minimum: 1 B/s
2024-06-13 12:30:59.227219  [DEBUG] current throughput: 0 B/s is below minimum: 1 B/s
2024-06-13 12:30:59.227223  [DEBUG] current throughput: 0 B/s is below minimum: 1 B/s
2024-06-13 12:30:59.227224  [DEBUG] current throughput: 0 B/s is below minimum: 1 B/s
2024-06-13 12:30:59.227228  [DEBUG] entering 'after deserialization' phase
2024-06-13 12:30:59.22723   [DEBUG] received a transient error, poisoning the connection...
2024-06-13 12:30:59.227233  [INFO] smithy connection was poisoned
2024-06-13 12:30:59.227234  [DEBUG] connection was poisoned poison_pill=PoisonPill@0x14301df40 { poisoned: true }
2024-06-13 12:30:59.227235  [DEBUG] the connection was poisoned
2024-06-13 12:30:59.227236  [DEBUG] finally_attempt;
2024-06-13 12:30:59.227238  [DEBUG] not retrying because we are out of attempts attempts=1 max_attempts=1
2024-06-13 12:30:59.227239  [DEBUG] a retry is either unnecessary or not possible, exiting attempt loop
2024-06-13 12:30:59.227239  [DEBUG] finally_op;
2024-06-13 12:30:59.227265  [WARN] [src/host/aws.rs:64] failed to invoke client in region ap-southeast-4: ResponseError(ResponseError { source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, raw: Response { status: StatusCode(200), headers: Headers { headers: {"date": HeaderValue { _private: H0("Thu, 13 Jun 2024 07:00:52 GMT") }, "content-type": HeaderValue { _private: H0("application/json") }, "content-length": HeaderValue { _private: H0("285818") }, "x-amzn-requestid": HeaderValue { _private: H0("64b98cc1-865e-4db7-b53c-1104ca412ef5") }, "x-amzn-remapped-content-length": HeaderValue { _private: H0("0") }, "x-amz-executed-version": HeaderValue { _private: H0("$LATEST") }, "x-amzn-trace-id": HeaderValue { _private: H0("root=1-666a991d-7c4170f35fb3a2ed78071a22;parent=617661a0b3eb4310;sampled=0;lineage=d58ac053:0") }} }, body: SdkBody { inner: Taken, retryable: false }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } })
Velfi commented 2 weeks ago

@wafflespeanut thanks for reporting this bug. I created a PR to fix it.

ysaito1001 commented 2 weeks ago

We see that the issue is during lambda invocation, but to be sure, could you also share trace logs that show the service name and operation name that run into this issue? An example line would look like

DEBUG invoke{service=s3 operation=GetObject sdk_invocation_id=8555156} ...
wafflespeanut commented 2 weeks ago

I do find a debug log line that looks like this:

[DEBUG] invoke; service=lambda operation=Invoke sdk_invocation_id=8300105

Do you want me to paste all log lines surrounding this as well?

ysaito1001 commented 2 weeks ago

Do you want me to paste all log lines surrounding this as well?

No, what you provided with us looks good. Thank you!

ysaito1001 commented 1 week ago

Disabling StalledStreamProtection for Lambda was released yesterday.

github-actions[bot] commented 1 week ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.