search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
2.98k stars 246 forks source link

Upgrade to `aws-sdk-sqs="1.23.0"` causing timeouts #1148

Closed Smotrov closed 4 months ago

Smotrov commented 4 months ago

Describe the bug

After updating to aws-sdk-sqs = "1.23.0" facing massive timeouts in the log.

Expected Behavior

Would be good if there will be no timeouts.

Current Behavior

Vast majority of calls to SQS are time outed with following error

[ERROR] SQS processor failed: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })

Reproduction Steps

Upgrade to aws-sdk-sqs = "1.23.0" Try to work with AWS SQS

Possible Solution

No response

Additional Information/Context

aws-sdk-sqs = "1.24.0" does not solving the bug

Version

➜  cargo_workspace git:(main) ✗ cargo tree | grep aws-
├── aws-config v1.3.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.6.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.9
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-sso v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-ssooidc v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-sts v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-apigateway v1.22.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-athena v1.23.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-cloudwatchlogs v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-dynamodb v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-s3 v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-sigv4 v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-sns v1.23.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-query v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-sqs v1.24.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-ssm v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-dynamodb v1.25.0 (*)
│   ├── aws-sdk-dynamodbstreams v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)


### Environment details (OS name and version, etc.)

Linux

### Logs

_No response_
Turbo87 commented 4 months ago

FWIW after merging https://github.com/rust-lang/crates.io/pull/8610 we noticed similar issues on the crates.io staging environment:

Failed to receive SQS queue message: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed

aajtodd commented 4 months ago

Thanks for the issue, we are looking into this now and how best to address it.

As a workaround you can disable upload throughput protection:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(
            StalledStreamProtectionConfig::enabled()
                .upload_enabled(false)
                .build(),
        )
        .load()
        .await;

This will keep stalled stream protection for downloads enabled and only disable uploads (which was the default prior to BehaviorVersion::v2024_03_28()). If you want to disable it completely for uploads and downloads:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(StalledStreamProtectionConfig::disabled())
        .load()
        .await;
kolloch commented 4 months ago

Explicitly setting a deprecated behavior version fixes this as well:

BehaviorVersion::v2023_11_09() instead of BehaviorVersion::v2024_03_28()

Smotrov commented 4 months ago

It seams like the fix is already merged into main? Which crate version includes it?

Velfi commented 4 months ago

We just released the runtime crate updates: https://github.com/smithy-lang/smithy-rs/releases/tag/release-2024-05-21 Once a release for the SDK crates runs later today (in about two hours) they'll include this fix.

Smotrov commented 4 months ago

New release brought new errors. Unfortunately unable so far... When getting S3 metadata: 

ResponseError(ResponseError { source: ContentLengthError { expected: 54736384, received: 0 }, raw: Response { status: StatusCode(200), headers: Headers { headers: {"x-amz-id-2": HeaderValue { _private: H0("uxo9Lk8l/KkksF3/QH5tFj1Ec4WiCO1yUKj/LrEdZPxeQ925zZFxTfJVqtDXhXnFou62IvgyKZWJH9QejfvyePXmf5qnXx/4") }, "x-amz-request-id": HeaderValue { _private: H0("S5Q8M1H1YPBS44KG") }, "date": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:57 GMT") }, "last-modified": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:53 GMT") }, "x-amz-expiration": HeaderValue { _private: H0("expiry-date=\"Fri, 24 May 2024 00:00:00 GMT\", rule-id=\"delete-unprocessed-files\"") }, "etag": HeaderValue { _private: H0("\"749e7afbd0311a630201cb61d439730e\"") }, "x-amz-server-side-encryption": HeaderValue { _private: H0("AES256") }, "x-amz-meta-pcc": HeaderValue { _private: H0("YYY") }, "x-amz-meta-pos": HeaderValue { _private: H0("US") }, "x-amz-meta-partition": HeaderValue { _private: H0("filtered") }, "accept-ranges": HeaderValue { _private: H0("bytes") }, "content-type": HeaderValue { _private: H0("application/octet-stream") }, "server": HeaderValue { _private: H0("AmazonS3") }, "content-length": HeaderValue { _private: H0("54736384") }} }, body: SdkBody { inner: Taken, retryable: false }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } }) raw response Some(Response { status: StatusCode(200), headers: Headers { headers: {"x-amz-id-2": HeaderValue { _private: H0("uxo9Lk8l/KkksF3/QH5tFj1Ec4WiCO1yUKj/LrEdZPxeQ925zZFxTfJVqtDXhXnFou62IvgyKZWJH9QejfvyePXmf5qnXx/4") }, "x-amz-request-id": HeaderValue { _private: H0("S5Q8M1H1YPBS44KG") }, "date": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:57 GMT") }, "last-modified": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:53 GMT") }, "x-amz-expiration": HeaderValue { _private: H0("expiry-date=\"Fri, 24 May 2024 00:00:00 GMT\", rule-id=\"delete-unprocessed-files\"") }, "etag": HeaderValue { _private: H0("\"749e7afbd0311a630201cb61d439730e\"") }, "x-amz-server-side-encryption": HeaderValue { _private: H0("AES256") }, "x-amz-meta-pcc": HeaderValue { _private: H0("YYY") }, "x-amz-meta-pos": HeaderValue { _private: H0("US") }, "x-amz-meta-partition": HeaderValue { _private: H0("filtered") }, "accept-ranges": HeaderValue { _private: H0("bytes") }, "content-type": HeaderValue { _private: H0("application/octet-stream") }, "server": HeaderValue { _private: H0("AmazonS3") }, "content-length": HeaderValue { _private: H0("54736384") }} }, body: SdkBody { inner: Taken, retryable: false }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } })
Velfi commented 4 months ago

I'm looking into a fix right now. Hopefully we can get it out this week.

ysaito1001 commented 4 months ago

@Smotrov, We have just released a new version in the smithy-rs repository. Curious to know whether the issue can be resolved if you use aws-smithy-runtime version 1.5.4.

Smotrov commented 4 months ago

Dear @Velfi, @aajtodd than you for you personal involvement. It seams like the issue is solved now 🎉

github-actions[bot] commented 4 months ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.