Use multiprocessing to speed up download time of solar data

OpenEnergyPlatform / data-preprocessing

Repository for data formatting, import of data, data and metadata review, and data curation.

GNU Affero General Public License v3.0

10 stars 7 forks source link

Use multiprocessing to speed up download time of solar data #19

Closed Bachibouzouk closed 5 years ago

Bachibouzouk commented 5 years ago

Maybe one can make use of multiprocessing also for the other download scripts?

Bachibouzouk commented 5 years ago

This video (and this repo : https://github.com/jreese/aiomultiprocess) might explain a way to do the job without too much concurrency but with substantial speed up.

Bachibouzouk commented 5 years ago

This was better addressed by @solar-c (unfortunately there is no PR number to refer to, the first commit was 2a95b209ac2b97a75b18066b88e9392e882f7cf3)