wri / gfw-raster-analysis-lambda

GFW Raster Analysis in AWS Lambda
8 stars 4 forks source link

Read all rasters in parallel upfront #16

Closed jterry64 closed 5 years ago

jterry64 commented 5 years ago

After some initial performance tests with XRay, I can see unsurprisingly IO consumes the most time. Currently all the rasters are being read one at time. One easy optimization we could do is just to read all the input rasters in parallel upfront.

A simple experiment on a 1 MHa geometry using the threading module showed it works pretty well. Before, each read took ~2 seconds, one after another. Multithreaded, as expected, it takes ~2 seconds total to read all at the same time.

jterry64 commented 5 years ago

1a5bf745cb1d5d2b4e98ac1bf7cbdd4f5cc9bde3 and 90b3db3bb63be90bd770ca75c20b9df8416a1c22