show statistics when used as library

[Fake-User]

Wow, what a killer project! The statistics in the CLI are really helpful. Is there a way to show the same statistics when using the package a in a rust project?

[nidor1998]

Thanks for your question.

Is there a way to show the same statistics when using the package a in a rust project?

Yes, you can use the pipeline.get_stats_receiver(). The returned receiver is a Receiver<SyncStatistics>. You can receive the statistics from the receiver.

The following is a sample program that shows the total number of synchronized objects. This is not a live statistics viewer. This receiver frequently receives statistics and store in the memory. So, you should receive the statistics frequently to avoid memory consumption. You may need to receive the statistics in a separate task.

use s3sync::config::args::parse_from_args;
use s3sync::config::Config;
use s3sync::pipeline::Pipeline;
use s3sync::types::token::create_pipeline_cancellation_token;
use s3sync::types::SyncStatistics;

#[tokio::main]
async fn main() {
    let args = vec![
        "sample_program",
        "./testdata_small",
        "s3://xxxx/src/",
    ];

    let config = Config::try_from(parse_from_args(args).unwrap()).unwrap();

    let cancellation_token = create_pipeline_cancellation_token();
    let mut pipeline = Pipeline::new(config.clone(), cancellation_token).await;

    // You can close statistics sender to stop statistics collection, if needed.
    // Statistics collection consumes some Memory, so it is recommended to close it if you don't need it.
    // pipeline.close_stats_sender();
    let stats_receiver = pipeline.get_stats_receiver();

    pipeline.run().await;
    assert!(!pipeline.has_error());

    let mut total_sync_count = 0;
    while let Ok(sync_stats) = stats_receiver.try_recv() {
        if matches!(sync_stats, SyncStatistics::SyncComplete { .. }) {
            total_sync_count += 1;
        }
    }

    println!("Total sync count: {}", total_sync_count)
}

If you want to implement a live statistics viewer, you can refer to the bin/cli/mod.rs and bin/cli/indicator.rs in the s3sync source code.

s3sync CLI is a very simple wrapper of the s3sync library. You can refer to the source code bin/cli to implement your own live statistics viewer.

[Fake-User]

so clutch, thanks mate