search

vladak / flickrknob

yet another Flickr uploader
0 stars 1 forks source link

parallelize upload #4

Closed vladak closed 3 years ago

vladak commented 3 years ago

When uploading a large set of files, there is often a pause between individual files in terms of network bandwidth, likely because the API is processing the file. To speed the overall process, there could be a pool of workers, each uploading one file at a time. Perhaps the workers can each have a sleep period initially to create sort of pipeline effect.

vladak commented 3 years ago

The sorting of the files in the album would have to be adapted - currently it is done before the upload. https://www.flickr.com/services/api/flickr.photosets.reorderPhotos.html would do the job it seems.

vladak commented 3 years ago

multiprocessing pool cannot be used because bar cannot be pickled. concurrent.future feels better anyway.