search

rigoudyg / climaf

CliMAF - a Climate Model Analysis Framework - doc at : http://climaf.readthedocs.org/
Other
18 stars 7 forks source link

Iterating operators on ensemble members should be parallel #24

Open senesis opened 9 years ago

senesis commented 9 years ago

See useful material on this subject at a link profvided by Sébastien : http://chriskiehl.com/article/parallelism-in-one-line/

jservonnat commented 6 years ago

I take this issue up. Recently I've been facing issues with heavy treatments on CORDEX EUR-11 outputs and it sounds like parallel processing could be of great help. Something like a cfile(my_ens, parallel=True), or a general setting do_parallel=True could be nice. In the short term we could use a function like:

def parallel_cfile(ens, nprocs=None):
     # -- Load libraries
     import multiprocessing
     from joblib import Parallel, delayed
     # -- Create list to store the objects we will execute in parallel
     crs_list = []
     for elt in ens: crs_list.append(ens[elt])
     # -- Get the number of processors
     if not nprocs: nprocs = multiprocessing.cpu_count()
     # -- Execute cfile on elements climaf_crs of list crs_list in parallel
     Parallel(n_jobs=nprocs)(delayed(cfile)(climaf_crs) for climaf_crs in crs_list)
     # -- Update CliMAF cache
     csync(True)
     # -- And return the result of cfile(ens)
     return cfile(ens)