awslabs / aws-s3-transfer-manager-rs

Apache License 2.0
5 stars 0 forks source link

Every setting is configurable per upload/download. #32

Open graebm opened 2 months ago

graebm commented 2 months ago

If this is going to sit under other systems someday, it must be possible for 2 downloads to be happening at the same time, each with totally different settings (e.g. different buckets, different regions, different part sizes).

aws-c-s3 started with settings such as part-size living on the s3_client. When it was integrated into other pre-existing transfer managers, configured part-size per-object. Eventually, all these settings had to be made configurable per download/upload.

Seems like there are 3 approaches we can take here, using part_size as an example:

There might be exceptions, for settings that are more "global" like max-memory-usage, max-bandwidth, etc

aajtodd commented 2 months ago

Not sure which way we want to go but the SDK uses the last option.

Common settings (retries, http client, etc) are set on the client but can be overridden per operation.

use aws_config::{BehaviorVersion, Region};

let config = aws_config::defaults(BehaviorVersion::latest())
    .region("us-east-1")
    .load()
    .await;

let s3 = aws_sdk_s3::Client::new(&config);

// Will be sent to "us-east-1"
s3.list_buckets()
    .send()
    .await?;

// Unset fields will default to using the original config value
let modified = aws_sdk_s3::Config::builder()
    .region(Region::from_static("us-west-2"));

// Will be sent to "us-west-2"
s3.list_buckets()
     // Creates a CustomizableOperation
    .customize()
    .config_override(modified)
    .send()
    .await?;