search

NNPDF / reportengine

A framework for declarative data analysis
https://data.nnpdf.science/validphys-docs/guide.html
GNU General Public License v2.0
1 stars 2 forks source link

Dask for parallel execution of the dag #56

Closed comane closed 1 year ago

comane commented 1 year ago

This PR makes use of dask.distributed for a parallel evaluation of the dag generated by reportengine.ResourceBuilder.

Zaharid commented 1 year ago

Super cool :)

I'll add more detailed comments,but seems it is down to details like the handling of the plotting style.

Zaharid commented 1 year ago

As discussed, I'd prefer it if the figuregen decorator didn't modify the original function when used outside reportengine, and kept it a generator. Then again, that would be a fairly niche problem, so might do later. Also I believe the main motivation for not doing lists as in here was not having too many "open" matplotlib (pylot) figures, which does not apply anymore.

The reason is mainly of consistency: At the moment reportengine tries to not disturb the user defined functions when it is not touching it, and the various decorators add attributes instead of modifying functionality.

Zaharid commented 1 year ago

@comane I am fine with this now. Feel free to merge!