search

trisongz / aws-sdk

Async boto3 with Autogenerated Data Classes
MIT License
1 stars 0 forks source link

async s3 sync #1

Open colin-ryan-airm opened 2 years ago

colin-ryan-airm commented 2 years ago

Hello, Not an issue exactly but I'm wondering if I can pick your brain on how I might use this (or what other options there are ) for fetching s3 bucket data in an async manner. My team is using aws s3 sync with --exclude to get sub sets of folders in buckets but it's quite a lot of data and I'm wondering if this lib could help in some way but I'm not sure how I would use it. Can you give me any guidance?

trisongz commented 2 years ago

Syncing is a tricky problem, and I think for ease of what you're trying to do, (since you said you're using the aws cli) I would honestly recommend going with gsutil or rsync. Personally, I prefer gsutil since you can use regex patterns.