SciQLop / speasy

Space Physics made EASY! A simple Python package to deal with main Space Physics WebServices (CDA,SSC,AMDA,..)
Other
24 stars 7 forks source link

Download job manager #37

Open jeandet opened 2 years ago

jeandet commented 2 years ago

While working on Catalog to DataFrame conv, I was thinking that maybe we could add a new module to easy scheduling data download. For example when an user want to retrieve a list of products data for each event in a Catalog, he has to do something like:

data = { product:[spz.get_data(product,*event) for event in catalog] for product in product_list}

Which is OK for the simplicity but lacks some control over parallelism.

We could provide something like:

data = speasy.jobs.get_data( product_list, catalogs, max_concurrent_jobs)