search

wellcomecollection / editorial-photography-ingest

0 stars 0 forks source link

Parallelise downloads #11

Closed paul-butcher closed 4 months ago

paul-butcher commented 4 months ago

In the first cut of this project, I ran the downloads sequentially. This seemed to be a bit slow (running locally) when I first pulled a semi-real shoot (EP-OPTEST, with actual tiffs, rather than my own test folder with some tiny screenshots).

See https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-parallel-reads-of-s3-objects-by-using-python-in-an-aws-lambda-function.html

paul-butcher commented 4 months ago

I have slightly broken my own rules by doing this now (make it work, make it right, make it fast - in order)