search

cloudyr / aws.s3

Amazon Simple Storage Service (S3) API Client
https://cloud.r-project.org/package=aws.s3
381 stars 147 forks source link

save_object() for multiple files #360

Open realauggieheschmeyer opened 4 years ago

realauggieheschmeyer commented 4 years ago

I have a question about package functionality.

Is it possible to use save_object to save multiple files at one time? I am trying to extract hundreds of photos from an S3 bucket and wasn't sure if I could use save_object to save these photos in batches.

An example of what I would like to do is:

save_object(c("file1.jpeg", "file2.jpeg", "file3.jpeg"), bucket = "some-bucket")

So far, I've been using a combination of map and save_object:

c("file1.jpeg", "file2.jpeg", "file3.jpeg") %>%
    map(save_object, bucket = "some_bucket")

This system works, but since it has to go file by file, it's a rather slow process. I was wondering if there was a faster way to achieve the same goal.

Thanks!

s-u commented 4 years ago

Short answer, sorry, no.

Long answer: it would make a lot of sense, since we could use parallel connections to make it a lot faster. However, that would require re-write of the request mechanism. It is underway in the https://github.com/cloudyr/aws.s3/tree/unify-response branch, so there is hope. I'll keep this open and hopefully we'll have a solution in the foreseeable future.

realauggieheschmeyer commented 4 years ago

Thanks a lot @s-u. I appreciate the consideration.