mojodna / marblecutter

Dynamic tiling of raster data for OpenAerialMap + others
Other
121 stars 25 forks source link

Create boto session once per thread #89

Open iandees opened 6 years ago

iandees commented 6 years ago

In render_pyramid.py I am creating one boto3 session per tile in order to handle the fact that boto3's session is not threadsafe. https://github.com/mojodna/marblecutter/blob/tilezen/examples/render_pyramid.py#L193

I was working on code to copy tiles out of a md5-hash prefixed bucket to a "regular" bucket and noticed that creating a session once per tile was eating up a lot of time, so instead I used a thread.local to store the boto session and initialized it once in the initializer for the Pool.

This took 75%+ off the time to do the simple copy operation. It might not take quite as much time off the pyramid renderer because there's more time spent on the actual rendering, but it would still be useful to do in the rendering scripts.

nvkelso commented 6 years ago

This is part of bulk renderer. This goes away using the MBTiles route.