search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
2.91k stars 245 forks source link

Hyper dns error when using docker #1138

Closed ripls56 closed 1 month ago

ripls56 commented 1 month ago

Describe the bug

Good afternoon, when trying to connect by url address, if I use docker I get an error related to dns.

Expected Behavior

Url address is recognised and I can access the storage.

Current Behavior

Url address is not recognised

Error:

DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })), connection: Unknown } })

Reproduction Steps

compose.yaml

services:
  minio:
    image: quay.io/minio/minio
    restart: always
    command: server /data --console-address ":9001"
    ports:
      - "12200:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: root
      MINIO_ROOT_PASSWORD: password
    networks:
      - my-network

  server:
    restart: always
    build:
      context: .
    depends_on:
      - minio
    ports:
      - 50055:1000
    networks:
      - my-network

networks:
  my-network:

main.rs

use aws_config::meta::region::RegionProviderChain;
use aws_config::Region;
use aws_sdk_s3::config::Credentials;
use aws_sdk_s3::Client;

#[tokio::main]
async fn main() -> Result<(), aws_sdk_s3::Error> {
    dotenv::dotenv().ok();

    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::ERROR)
        .init();

    let region = "eu";
    let endpoint = "http://minio:9000";
    let access_key = "root";
    let secret_key = "password";

    let cred_provider = Credentials::new(access_key, secret_key, None, None, "minio");
    let region_provider = RegionProviderChain::first_try(Region::new(region));
    let config = aws_config::from_env()
        .credentials_provider(cred_provider)
        .region(region_provider)
        .load()
        .await;

    let s3_config = aws_sdk_s3::config::Builder::from(&config)
        .endpoint_url(endpoint)
        .build();

    let client = Client::from_conf(s3_config);

    let bucket_name = "test-bucket";

    client
        .create_bucket()
        .bucket(bucket_name)
        .send()
        .await
        .unwrap();

    Ok(())
}

Cargo.toml

[package]
name = "s3"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
dotenv = "0.15.0"
aws-config = { version = "*", features = ["behavior-version-latest"] }
aws-sdk-s3 = "*"
tokio = { version = "*", features = ["full"] }
tracing-subscriber = "0.3.18"
tracing = "0.1.40"

Possible Solution

No response

Additional Information/Context

I also tried localstack to exclude any problems with minio.

However, it works if you access the storage outside the docker network (i.e. only the storage is deployed)

Also I am confused that the access to the bucket goes through the sub domain (attached in the logs), I am a beginner so I don't know how it should be 😔

Version

├── aws-config v1.2.1
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.4.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.8
│   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-runtime v1.2.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.2.0
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-sdk-sso v1.21.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.0 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime v1.3.1
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.2.0 (*)
│   ├── aws-sdk-ssooidc v1.21.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.0 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.3.1 (*)
│   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.2.0 (*)
│   ├── aws-sdk-sts v1.21.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.0 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime v1.3.1 (*)
│   │   ├── aws-smithy-runtime-api v1.4.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.0 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.3.1 (*)
│   ├── aws-smithy-runtime-api v1.4.0 (*)
│   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-types v1.2.0 (*)
├── aws-sdk-s3 v1.24.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.0 (*)
│   ├── aws-sigv4 v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.3.1 (*)
│   ├── aws-smithy-runtime-api v1.4.0 (*)
│   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.0 (*)

Environment details (OS name and version, etc.)

macOS 14.4.1 23E224 darwin-arm64

Logs

server-1  | 2024-05-03T00:09:13.799235Z TRACE invoke{service=s3 operation=CreateBucket}:try_op:try_attempt: hyper::client::pool: checkout waiting for idle connection: ("http", test-bucket.minio:9000)
server-1  | 2024-05-03T00:09:13.799322Z TRACE invoke{service=s3 operation=CreateBucket}:try_op:try_attempt: hyper::client::connect::http: Http::connect; scheme=Some("http"), host=Some("test-bucket.minio"), port=Some(Port(9000))
server-1  | 2024-05-03T00:09:13.799385Z DEBUG hyper::client::connect::dns: resolving host="test-bucket.minio"
brainstorm commented 1 month ago

Hello @ripls56, it turns out we struggled with this. MINIO_DOMAIN was not set, see this PR: https://github.com/umccr/htsget-rs/pull/232/files

TL;DR it turns out that MinIO supports S3's path-style by default (which AWS will deprecate) whereas the S3 sdk goes for virtual-hosted-style instead by default and that's where you see those DNS errors.

So you might want to force path_style like so. Also note that your particular local DNS resolver (OS level) and docker listen/host configuration (i.e: listening to 0.0.0.0 or not?) might also play a role here.

Hope that helps!

ripls56 commented 1 month ago

Hi @brainstorm, thanks for your help. Everything is working now, but am I right in understanding that I can't create buckets at runtime because there will be no associated names for them in the docker? (if I don't use path_style)

github-actions[bot] commented 1 month ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

brainstorm commented 1 month ago

Hi @brainstorm, thanks for your help. Everything is working now, but am I right in understanding that I can't create buckets at runtime because there will be no associated names for them in the docker? (if I don't use path_style)

I haven't tested that particular bit since our buckets are pre-created... but I'd guess you can create them at runtime, you just have to adjust the style/how to reach them? Fire up a wireshark/tshark packet inspector and you'll probably see DNS requests failing and then you can fix it up from there.