search

awslabs / aws-s3-transfer-manager-rs

Apache License 2.0
5 stars 0 forks source link

Crate organization #35

Closed aajtodd closed 2 months ago

aajtodd commented 2 months ago

Tracking issue to re-organize crate with some kind of defined structure with intent and purpose.

aajtodd commented 2 months ago

Strawman proposal:

// relative to src/

lib.rs
client.rs               // Transfer manager client
error.rs                // common errors/re-exports
types.rs                // shared types for operations/configuration

client/                 // client related support
    builder.rs
    config.rs           // client configuration
    config/
        builder.rs
        loader.rs
        ...

    handle.rs           // Shared handle to configuration and runtime components re-used across operations (e.g. config, scheduler, task executor, etc) 

    uploader/           // uploader implementation
        handle.rs
        context.rs
        error.rs
        builder.rs
        ...

    downloader/         // downloader implementation
        handle.rs
        context.rs
        error.rs
        builder.rs
        ...

io/                     // io/network related utils

fs/                     // filesystem related utils

rt/                     // runtime related utils (scheduling/budgets/task execution/etc)
    scheduler.rs        // global scheduling
    memory_limit.rs 
    ...

operation/              // operation input/output/builders and orchestration
    upload.rs.          

    upload/
        request.rs
        response.rs
        builders.rs

    upload_dir.rs
    upload_dir/
        ...

    download.rs
    download_objects.rs
    ...

NOTE: uploader/downloader are lower level clients for upload and download respectively. Transfer manager client is the higher level SEP abstraction that glues it all together in a sensible way. We may choose to expose Uploader and Downloader clients for individual use IF it makes sense. An uploader/downloader will take shared runtime components (like scheduler, etc) which will be used to "globally" coordinate concurrency, memory usage, etc.

use aws_s3_transfer_manager::client::Uploader;   // uploader
use aws_s3_transfer_manager::client::Downloader; // downloader

use aws_s3_transfer_manager::Client;             // Transfer manager
aajtodd commented 2 months ago

One change to what was proposed. For now we are going to move downloader and uploader internals into the download and upload operations respectively. Uploader and Downloader were meant as "low level" clients used to upload/download an object as fast as possible as an implementation detail of the higher level transfer manager client. We are choosing to not expose these for now and as such can just remove them as concepts and make it an implementation detail of the singular upload and download operations. The other operations that involve uploading and dowloading multiple objects will just re-use the singular operations (and/or internals). This is slightly different from the AWS SDK where operations don't depend on one another. As a HLL this seems fine.