search

coursera / dataduct

DataPipeline for humans.
Other
252 stars 83 forks source link

Boto multipart upload #191

Closed darinyu-coursera closed 8 years ago

sb2nov commented 8 years ago

Let's measure the upload times before we roll this out.

darinyu-coursera commented 8 years ago 
0%                100%
[####################] | ETA[sec]: 0.000
Total time elapsed: 656.989 sec

This is what progress bar will look like.

The performance wasn't boosted as much as I expect. The upload time are still roughly the same. I am not sure whether I should use multi-process to make this run faster.

sb2nov commented 8 years ago

I think we need to use multiprocessing here over the files array so that we can make this faster.

sb2nov commented 8 years ago

@darinyu-coursera I'm going to close this as there is if we don't see performance gains might not be worth making stuff more complicated.