SIMEXP / Repo2Data

Automatic data fetcher from the web
MIT License
7 stars 3 forks source link

Feature request : multi-threading for data download #9

Open ltetrel opened 5 years ago

ltetrel commented 5 years ago

I should do multi threading in case of multiple requirement files. We do not want multi processing because we are not CPU bounded but I/O bounded in this case, so GIL is not a problem here. And multi threading is easier than multiprocessing because data is shared between threads.