search

cboudereau / gcs-rsync

Lightweight Google Cloud Storage sync Rust Client with better performance than gsutil rsync
https://docs.rs/gcs-rsync/
MIT License
13 stars 8 forks source link

rsync from a list of files #17

Closed bernardohenz closed 1 year ago

bernardohenz commented 2 years ago

Hi,

I have a huuge dataset and I only want to sync some of the files. The traditional rsync allows for that (using --files-from), but the rsync on gsutil does not have this functionality.

Does your implementation allow such a thing? If not, can this be implemented?

cboudereau commented 2 years ago

Hi! I am actually on holidays.

It is totally possible to add a filter.

A good start could be to add a function to the builder to add the filter like the restore fs time and update the sync function by applying the filter after the list() call.

On Tue, Aug 9, 2022, 13:32 Bernardo @.***> wrote:

Hi,

I have a huuge dataset and I only want to sync some of the files. The traditional rsync allows for that (using --files-from), but the rsync on gsutil does not have this functionality.

Does your implementation allow such a thing? If not, can this be implemented?

— Reply to this email directly, view it on GitHub https://github.com/cboudereau/gcs-rsync/issues/17, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP6WVZKHPHXRGRAM6UXDMTVYI6TNANCNFSM56AN7X3A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

cboudereau commented 2 years ago

While I am implementing the feature, there is might be an example that could fit with your need: if you already now which files you have to sync, you could use this example

cboudereau commented 1 year ago

Hi @bernardohenz, sorry for the huge delay.

I am implementing the feature in #19.

I have found a reasonable trade-offs by adding --include and --exclude from the CLI which can be filled multiple times.

The library has also been updated. I don't want to add the --files-from before adding include and exclude glob pattern.

cboudereau commented 1 year ago

@bernardohenz, the new version is published (v0.3.1) which supports include and exclude glob patterns as described in the pr #19 👍

cboudereau commented 1 year ago

I am gonna close the issue.

As I mentioned, we will introduce exclusion from a file later since the exclusion argument can be repeated.