awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
3.02k stars 248 forks source link

Use /etc/hosts binding, some APIs are not available #1187

Closed jiashiwen closed 1 month ago

jiashiwen commented 1 month ago

Describe the bug

When using /etc/hosts for binding, some APIs report errors. List_buckets, list_objects_v2, and get_object are verified here.

Expected Behavior

get bucket list ok,get objectlist ok,get object ok

Current Behavior

got bucket ok ,but error got objects list and get object

got buckets: Ok( ListBucketsOutput { buckets: Some( [ Bucket { name: Some( "ai-datasets", ), creation_date: Some( 2024-02-05T02:28:04Z, ), }, Bucket { name: Some( "cs-monitor", ), creation_date: Some( 2023-02-07T10:12:00Z, ), }, Bucket { name: Some( "dataroom", ), creation_date: Some( 2024-05-16T10:26:42Z, ), }, Bucket { name: Some( "hackthon2023", ), creation_date: Some( 2023-08-01T01:43:57Z, ), }, Bucket { name: Some( "jsw-bucket", ), creation_date: Some( 2023-02-23T03:11:46Z, ), }, Bucket { name: Some( "jsw-bucket-1", ), creation_date: Some( 2023-09-20T09:43:15Z, ), }, Bucket { name: Some( "pingdata", ), creation_date: Some( 2021-08-27T03:35:08Z, ), }, Bucket { name: Some( "robot-test", ), creation_date: Some( 2022-04-12T06:41:43Z, ), }, Bucket { name: Some( "robots", ), creation_date: Some( 2021-11-09T03:03:23Z, ), }, Bucket { name: Some( "tsp-picture", ), creation_date: Some( 2022-01-10T03:17:10Z, ), }, ], ), owner: Some( Owner { display_name: Some( "jcloud_edUmprJ", ), id: Some( "577257366345", ), }, ), continuation_token: None, _extended_request_id: None, _request_id: Some( "917104E844BBC1FA", ), }, ) objects: Err( DispatchFailure( DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error( Connect, ConnectError( "dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known", }, ), ), connection: Unknown, }, }, ), ) obj: Err( DispatchFailure( DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error( Connect, ConnectError( "dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known", }, ), ), connection: Unknown, }, }, ), )

Reproduction Steps

code

#[tokio::main]
async fn main() {
    let ak = "xxx";
    let sk = "xxx";
    let region = "cn-north-1";
    let endpoint = "http://xxx.cn-north-1.xxx.com";
    let bucket = "jsw-bucket";

    let config = SdkConfig::builder()
        .credentials_provider(SharedCredentialsProvider::new(Credentials::new(
            ak, sk, None, None, "Static",
        )))
        .endpoint_url(endpoint)
        .region(Region::new(region))
        .behavior_version(BehaviorVersion::v2024_03_28())
        .build();

    let s3_config_builder = aws_sdk_s3::config::Builder::from(&config);
    let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());

    let buckets = client.list_buckets().send().await;
    println!("got buckets: {:#?}", buckets);

    let list = client
        .list_objects_v2()
        .bucket("jsw-bucket")
        .max_keys(20)
        .send()
        .await;
    println!("objects: {:#?}", list);

    let obj = client
        .get_object()
        .bucket(bucket)
        .key("100k/!!41725505223551787799")
        .send()
        .await;
    println!("obj: {:#?}", obj);
}

/etc/hosts bind name

ip xxx.cn-north-1.xxx.com

Possible Solution

No response

Additional Information/Context

No response

Version

"B0E80D4FC4528926",
        ),
    },
)
root@a30:~/rustprojects/oss_pipe# cargo tree | grep aws-
├── aws-config v1.5.5 (/root/rustprojects/aws-sdk-rust/sdk/aws-config)
│   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api)
│   │   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-sigv4 v1.2.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-sigv4)
│   │   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstream)
│   │   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http)
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstrea
│   │   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api
│   │   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstream) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime)
│   │   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types)
│   │   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-sdk-sso v1.40.0 (/root/rustprojects/aws-sdk-rust/sdk/sso)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json)
│   │   │   └── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-sdk-ssooidc v1.41.0 (/root/rustprojects/aws-sdk-rust/sdk/ssooidc)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-sdk-sts v1.40.0 (/root/rustprojects/aws-sdk-rust/sdk/sts)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   │   ├── aws-smithy-query v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-query)
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-xml v0.60.8 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-xml)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
├── aws-sdk-s3 v1.47.0 (/root/rustprojects/aws-sdk-rust/sdk/s3)
│   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   ├── aws-sigv4 v1.2.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-sigv4) (*)
│   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-checksums v0.60.12 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-checksums)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstream) (*)
│   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-smithy-xml v0.60.8 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-xml) (*)
│   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)

Environment details (OS name and version, etc.)

ubuntu 22.04

Logs

No response

njaard commented 1 month ago

I'm using strois, which uses ureq, which uses rust's std DNS resolver, and this just started happening to me as well, but only if I do a bunch of requests simultaneously.

I don't have any /etc/hosts entries for AWS

ysaito1001 commented 1 month ago

Thank you for reporting the issue. Did it start occurring after you upgraded aws-sdk-s3 (and others) to the versions you pasted above? And if so, what was the last working version of aws-sdk-s3?

njaard commented 1 month ago

@ysaito1001 I'm not using aws-sdk-s3

Are you using cloudflare DNS (1.1.1.1)?

ysaito1001 commented 1 month ago

@njaard, Hmm, if the same issue occurs with strois and a different HTTP client, the issue could be outside our SDK.

ysaito1001 commented 1 month ago

@ysaito1001 I'm not using aws-sdk-s3

Are you using cloudflare DNS (1.1.1.1)?

No we are not, at least not that I know of.

njaard commented 1 month ago

Yes, I think the similarity in our problems is a coincidence.

jiashiwen commented 1 month ago

The problem is caused by request style, change to path style to solve the problem

let s3_config_builder = aws_sdk_s3::config::Builder::from(&config).force_path_style(true);
let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());

thanks!

github-actions[bot] commented 1 month ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.