search

nidor1998 / s3sync

Full featured, very fast s3 synchornization CLI tool powered by rust
Apache License 2.0
11 stars 1 forks source link
aws aws-s3 rust s3 s3-bucket

readme

s3sync

CI codecov DeepSource

Overview

s3sync is a reliable, very fast, and powerful synchronization tool for S3.
It can be used to synchronize local directories with S3 bucket, and also to synchronize S3 to s3 bucket. Supports multipart upload, versioning, metadata.

As a library

s3sync can be used as a library.

use s3sync::config::args::parse_from_args;
use s3sync::config::Config;
use s3sync::pipeline::Pipeline;
use s3sync::types::token::create_pipeline_cancellation_token;

#[tokio::main]
async fn main() {
    // You can use all the arguments for s3sync binary here.
    let args = vec!["program_name", "./src", "s3://test-bucket/src/"];

    // s3sync library converts the arguments to Config.
    let config = Config::try_from(parse_from_args(args).unwrap()).unwrap();

    // Create a cancellation token for the pipeline.
    // You can use this token to cancel the pipeline.
    let cancellation_token = create_pipeline_cancellation_token();
    let mut pipeline = Pipeline::new(config.clone(), cancellation_token).await;

    // You can close statistics sender to stop statistics collection, if needed.
    // Statistics collection consumes some Memory, so it is recommended to close it if you don't need it.
    pipeline.close_stats_sender();

    // Run the pipeline. In this simple example, we run the pipeline synchronously.
    pipeline.run().await;

    assert!(!pipeline.has_error());
}

Features

Requirements

Licence

This project is licensed under the Apache-2.0 License.

Installation

Download the latest binary from Releases
The binary runs on the above platforms without any dependencies.

You can also build from source following the instructions below.

Install Rust

See https://www.rust-lang.org/tools/install

Build

s3sync requires Rust 1.76 or later.

cargo install s3sync --path .

Usage

AWS credentials are required to use s3sync. IAM Roles, AWC CLI Profile, environment variables, etc supported.
By default s3sync access credentials from many locations(IAM Roles, environment variables, etc).
For more information, see Specifying your credentials and default region

You can also specify credentials in the command line options or environment variables.

Region is required. It can be specified in the profile, environment, or command line options.

Prefix is optional. If not specified, the entire bucket will be synchronized.

If you specify a prefix, s3sync doesn't automatically add trailing slash.
For example, if you specify s3://bucket-name/prefix as the target, s3sync will synchronize source objects named foo to s3://bucket-name/prefixfoo.
And if you specify s3://bucket-name/prefix/ as the target, s3sync will synchronize source objects named foo to s3://bucket-name/prefix/foo.

If you use s3sync for the first time, you should use the --dry-run option to test the operation.

For all options, see s3sync --help

Local to S3

s3sync /path/to/local s3://bucket-name/prefix

S3 to Local

s3sync s3://bucket-name/prefix /path/to/local

S3 to S3

s3sync s3://bucket-name1/prefix s3://bucket-name2/prefix

Access key and secret access key directly

s3sync --source-access-key foo --source-secret-access-key bar s3://bucket-name/prefix /path/to/local

AWS CLI profile support

s3sync --source-profile foo --target-profile bar s3://bucket-name1/prefix s3://bucket-name2/prefix

Specify region

s3sync --source-profile foo --source-region ap-northeast-1 s3://bucket-name1/prefix s3://bucket-name2/prefix

Versioning

s3sync --enable-versioning s3://bucket-name1/prefix s3://bucket-name2/prefix

Custom endpoint

You can specify a S3-compatible storage endpoint.
Warning: If you use a custom endpoint, you may need to specify --source-force-path-style or --target-force-path-style.

s3sync --target-endpoint-url https://foo --target-force-path-style /path/to/local s3://bucket-name/prefix

Dry run

s3sync --dry-run s3://bucket-name1/prefix s3://bucket-name2/prefix

--delete

s3sync --delete s3://bucket-name1/prefix s3://bucket-name2/prefix

About end-to-end object integrity check

s3sync calculates ETag(MD5) checksums for source object and compares them with the checksums in the target.
Optionally, s3sync can also calculate and compare additional checksum(SHA256/SHA1/CRC32/CRC32C) for each object.

ETag is not always MD5. If the object is uploaded with multipart upload, ETag is not MD5 digest but the MD5 digest of these concatenated values.
Even if multipart upload is used, s3sync can calculate and compare ETag for each part and the entire object.(with --auto-chunksize)

If the object is uploaded with SSE-KMS/SSE-C, ETag is not MD5. In this case, s3sync cannot calculate and compare checksums for the object.

s3sync always uses the following elements to verify the integrity of the object.

The multipart ETag does not always match that of the source object. But with above elements, it is almost safe to assume that the object is not corrupted.
With --auto-chunksize, s3sync can calculate and compare ETag/checksum for each part and the entire object.
If you want strict integrity check, use --auto-chunksize, but it will need more API calls and time.

For more information, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

Matrix of s3sync end-to-end object integrity check

In the case of Local to S3 or S3 to Local, s3sync compares ETags and additional checksums of Local that s3sync calculates with those of S3.
In the case of S3 to S3, s3sync just compares ETags and additional checksums that calculated by S3.

If an object is not verified, s3sync show a warning message in the terminal.

Amazon S3 Express One Zone storage class does not support ETag as verification. But you can use additional checksum algorithm.

ETag(MD5 digest or equivalent): plain-text/SSE-S3

Local to S3 S3 to Local S3 to S3
multipart object always verify verify as much as possible(without --auto-chunksize)
always verify(with --auto-chunksize)		 verify as much as possible(without --auto-chunksize)
always verify(with --auto-chunksize)
single object always verify always verify always verify

ETag(MD5 digest or equivalent): SSE-KMS/SSE-C

Local to S3 S3 to Local S3 to S3
multipart object not verify not verify not verify
single object not verify not verify not verify

Additional checksum: plain-text/SSE-S3/SSE-KMS/SSE-C

Local to S3 S3 to Local S3 to S3
multipart object always verify always verify verify as much as possible (without --auto-chunksize)
always verify(with --auto-chunksize)
single object always verify always verify always verify

Note: To enable end-to-end additional checksum verification, --enable-additional-checksum must be specified.
Note: In the case of S3 to S3, same checksum algorithm must be used for both source and target.

About --auto-chunksize

If --auto-chunksize is specified, s3sync automatically calculates the correct chunk size for multipart upload.
This is done by HeadObject API with partNumber parameter. --auto-chunksize requires extra API calls(1 API call per part).
Remember that not all S3-compatible storage supports HeadObject API with partNumber parameter.
If S3-compatible storage does not support it, s3sync will show a warning message in the terminal.

See: https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_RequestSyntax

Warning: In case of S3 to S3, if the source object is uploaded with a large chunk size, s3sync will consume a lot of memory.

Incremental transfer

s3sync transfers only modified objects.It checks LastModified timestamp.

If --check-size is specified, s3sync only checks the size of the object.

At first, incremental transfer lists all objects in the target.
Then, s3sync compares the objects and transfers only modified objects.

Therefore, incremental transfer takes a time to start for large amounts of existing target objects.
In my environment, Amazon S3 takes about 1 second per 10,000 objects to list objects in the target.(It depends on remote storage.)

If there are few objects in the target, incremental transfer starts immediately, regardless of the number of source objects.

Versioning support

s3sync uses user-defined metadata to check the version of the object.
If --enable-versioning is specified, s3sync adds user-defined metadata to the object. If you transfer to existing bucket, because of the lack of user-defined metadata, s3sync will transfer all versions of the object.
Generally, you should use --enable-versioning when you transfer to a new bucket.

Intermediate delete markers are not synchronized. Latest version delete markers are synchronized.

user-defined metadata: s3sync_origin_version_id, s3sync_origin_last_modified

Metadata support

The following metadata of the S3 object is synchronized.

SSE support

The following SSE is supported.

Memory usage

s3sync consumes memory for each worker.
For single object, approximately average size of the object * worker-size(default 16) * 2.
For multipart object, approximately multipart chunksize(default 8MiB) * worker-size(default 16) * 2.

Because s3sync uses incremental transfer, it lists all objects in the target bucket and stores the result in memory.
Therefore, if there are a large number of objects in the target bucket, s3sync can consume a lot of memory.
If you do not use the --delete option, s3sync will consume about 100MB per 1,000,000 target objects.
If you use the --delete option, s3sync will consume about 250MB per 1,000,000 target objects.

To reduce memory usage, you can divide the target objects by prefix and run s3sync multiple times or specify --remove-modified-filter and --head-each-target options.
You can also divide the target objects by filter and run s3sync multiple times.

The --head-each-target option calls the HeadObject API for each source object, but it is a trade-off between memory usage and API calls.

S3 Permissions

s3sync requires the following permissions.

Source bucket

"Action": [
    "s3:GetBucketVersioning",
    "s3:GetObject",
    "s3:GetObjectAttributes",
    "s3:GetObjectTagging",
    "s3:GetObjectVersion",
    "s3:GetObjectVersionAttributes",
    "s3:GetObjectVersionTagging",
    "s3:ListBucket",
    "s3:ListBucketVersions",
    "s3express:CreateSession",
]

Target bucket

"Action": [
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:DeleteObjectTagging",
    "s3:GetBucketVersioning",
    "s3:GetObject",
    "s3:GetObjectTagging",
    "s3:GetObjectVersion",
    "s3:ListBucket",
    "s3:ListBucketVersions",
    "s3:PutObject",
    "s3:PutObjectTagging",
    "s3express:CreateSession"
]

Advanced options

--worker-size

The number of workers. Many workers can improve performance. But it also increases CPU and memory usage.
Default: 16

If you specify many workers, you may need to increase the number of open files.
For example, on Linux: ulimit -n 8192

--force-retry-count

s3sync forcibly retries the operation that AWS SDK for Rust cannot retry.
For example, in the case of connection reset by peer, s3sync will retry the operation.

--remove-modified-filter

If you want to overwrite the existing objects, specify the option.

--check-etag

For incremental transfer, s3sync compares the ETag of the source object with the ETag of the target object. If the ETag is different, s3sync transfers the object.

with --auto-chunksize, s3sync can calculate and compare ETag for each part and the entire object. It is useful if you don't know the correct chunk size. But it will need more API calls and time.

If both sides are S3, s3sync only compare the ETag of the source object with the ETag of the target object. If either side is not S3, s3sync calculates ETag of the local object.

You will need to know about Amazon S3 ETag.
See: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html

--put-last-modified-metadata

This option adds user-defined metadata that contains the last-modified time of the source object.

user-defined metadata: s3sync_origin_last_modified

--additional-checksum-algorithm

If you want to use additional checksum for upload, specify the algorithm.

--enable-additional-checksum

If you want to use additional checksums for download, specify the option.

Warning: Even if the object was uploaded with additional checksum, without this option, s3sync does not verify additional checksum.

--https-proxy

You can specify the proxy server for https.

Proxy authentication is supported. Like http(s)://user:password@proxy:port.

--disable-multipart-verify

When object is uploaded with multipart upload, its ETag may not match that of the target object.
This can occur when the chunk size that the object was uploaded with is different. If you don't know the correct chunk size, you can disable the verification with this option.
You can specify the chunk size with --multipart-threshold and --multipart-chunksize (Default: 8MiB).

If extra API calls are allowed, you can use --auto-chunksize instead.
However, please note that not all S3-compatible storage supports this option.
Warning: In case of S3 to S3, if the source object is uploaded with a large chunk size, s3sync will consume a lot of memory.

-v

s3sync uses tracing-subscriber for tracing.
More occurrences increase the verbosity.
For example, -v: show info, -vv: show debug, -vvv: show trace
By default, s3sync shows warning and error messages.

info and debug messages are useful for troubleshooting. trace messages are useful for debugging.

Instead of -v, you can use RUST_LOG environment variable.

--aws-sdk-tracing

For troubleshooting, s3sync can output the AWS SDK for Rust's tracing information.
Instead of --aws-sdk-tracing, you can use RUST_LOG environment variable.

--filter-include-regex, --filter-exclude-regex

You can specify the regular expression to filter the source objects.
The regular expression syntax is the same as regex.

--auto-complete-shell

You can output the shell script to complete the command.

s3sync --auto-complete-shell bash

-h/--help

For more information, see s3sync -h.