search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
2.91k stars 245 forks source link

Unable to perform multipart upload after upgrade: current throughput: 0 B/s is below minimum: 1 B/s #1146

Closed DimanNe closed 3 weeks ago

DimanNe commented 1 month ago

Describe the bug

After this upgrade:

aws-config   1.3.0 -> 1.4.0
aws-sdk-s3   1.25.0 -> 1.28.0
aws-smithy-types 1.1.8 -> 1.1.9

multi-part upload started to fail with the following messages in logs:

[2024-05-13T22:40:02.459500Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.459541Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.461266Z DEBUG hyper::proto::h1::dispatch] error writing: Connection reset by peer (os error 104)

[2024-05-13T22:40:02.661296Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.661314Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.664376Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] current throughput: 0 B/s is below minimum: 1 B/s
[2024-05-13T22:40:02.664381Z DEBUG aws_smithy_runtime::client::http::body::minimum_throughput] grace period ended; timing out request
[2024-05-13T22:40:02.664390Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting
[2024-05-13T22:40:02.664394Z DEBUG tracing::span] finally_attempt;
[2024-05-13T22:40:02.664409Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] not retrying because we are out of attempts attempts=3 max_attempts=3
[2024-05-13T22:40:02.664414Z DEBUG aws_smithy_runtime::client::orchestrator] a retry is either unnecessary or not possible, exiting attempt loop
[2024-05-13T22:40:02.664417Z DEBUG tracing::span] finally_op;

Same code works perfectly fine with the older version.

Expected Behavior

see above

Current Behavior

see above

Reproduction Steps

see above

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v1.4.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.6.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.9
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-sso v1.24.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-ssooidc v1.25.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-sts v1.24.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-s3 v1.28.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-sigv4 v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-smithy-types v1.1.9 (*)

Environment details (OS name and version, etc.)

Linux

Logs

No response

Smotrov commented 1 month ago

Same error with SQS

aajtodd commented 1 month ago

Thanks for the issue, we are looking into this now and how best to address it.

As a workaround you can disable upload throughput protection:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(
            StalledStreamProtectionConfig::enabled()
                .upload_enabled(false)
                .build(),
        )
        .load()
        .await;

This will keep stalled stream protection for downloads enabled and only disable uploads (which was the default prior to BehaviorVersion::v2024_03_28()). If you want to disable it completely for uploads and downloads:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(StalledStreamProtectionConfig::disabled())
        .load()
        .await;
Velfi commented 1 month ago

We just released the runtime crate updates: https://github.com/smithy-lang/smithy-rs/releases/tag/release-2024-05-21 Once a release for the SDK crates runs later today (in about two hours) they'll include this fix.

ysaito1001 commented 1 month ago

Let us know whether today's release has improved the behavior of your use case.

github-actions[bot] commented 3 weeks ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.