search

rubisco-sfa / ILAMB

Python software used in the International Land Model Benchmarking (ILAMB) project
BSD 3-Clause "New" or "Revised" License
46 stars 37 forks source link

Try to merge ilamb result #78

Open rhaegar325 opened 1 year ago

rhaegar325 commented 1 year ago

Can we merge ilamb result in multiple runs?

nocollier commented 1 year ago

The short answer is no, we have discussed implementing this but it has not risen to high enough priority as simply running the whole study again is an option. In my mind we would want to implement a 'results caching' type functionality but it is tricky as there are many aspects which might make an ilamb result unmergable: the ilamb code version, reference data version, configuration file, options used, etc..

The long answer is yes, you can manually do this if you understand how the ilamb-run program works. We first perform all the analysis and dump everything into the _build directory as intermediate netCDF files. The second pass walks through the results and opens all the netCDF files and then determines plot limits, and plots everything. So you could write a utility that walks one _build1 directory tree and copies netcdf files into another _build2 directory. Then run ilamb-run again on the _build2 and ilamb will regenerate the plots and webpages.

But just running the analysis again is cleaner. If you are running large studies and not using parallelism, I can help you with that. I run on a large memory node cluster here typically on ~30 processes over ~4 nodes and the full CMIP5v6 run completes in ~1 hour.