speed up tile download with concurrent processing

developmentseed / label-maker

Data Preparation for Satellite Machine Learning

http://devseed.com/label-maker/

MIT License

461 stars 110 forks source link

speed up tile download with concurrent processing #182

Closed martham93 closed 4 years ago

martham93 commented 4 years ago

to address issue #12

uses ThreadPoolExecutor to speed up tile downloads
previously downloading ~400 tiles from a mapbox endpoint took ~31 second, with using the threadpool executor downloading ~400 tiles takes ~8 seconds!
if this change aligns with what you were thinking, I think it's possible to do something similar with the child tile download for the supertiles special case, and I can investigate

cc @drewbo

drewbo commented 4 years ago

Re: supertile child download: that function will already be happening inside the ThreadPool (because download_tile_tms is one of the image_download options)

martham93 commented 4 years ago

the download time for ~400 tiles with thread count of 50 is ~4 seconds!

drewbo commented 4 years ago

@martham93 sweet; looks better. Last two things: (1) make the default threads ~10 and then people can up it if they want and (2) either remove the time elapsed or print it with nicer formatting

martham93 commented 4 years ago

awesome thanks @drewbo all of these should be addressed now