search

uaf-arctic-eco-modeling / dvm-dos-tem

A process based Dynamic Vegetation, Dynamic Organic Soil, Terrestrial Ecosystem Model.
MIT License
22 stars 24 forks source link

Calibration/optimization workflow, need standardized methods to "export" the calibrated parameters when optimization is done. #669

Open tobeycarman opened 10 months ago

tobeycarman commented 10 months ago

When the optimization is complete (using MADS), there are 2 files, .iterationresults and .finalresults that have the calibrated parameters. These parameters should be used as the starting values for the next steps of calibration or sensitivity work. Right now there are 2 ways this could be done, both of which are somewhat manual:

  1. There are several functions in the mads_calibration/utils.py file that deal with extracting the parameters and preparing them for plotting.
  2. There is a function, mads_calibration/SA_post_hoc_analysis.read_mads_iteration_results(...) that can extract data from these special text files and return a python data structure.

The user is then left to manually put these parameters into the directory that they would like to use to seed the next step. Historically this has been the /work/parameters directory (if user is working in the Docker container). This presents a problem because the parameter modifications show up as changes in the version control system, and this can make it difficult for user who are not comfortable with Git to handle the changes when they want to update from upstream.

Current thinking is that the user will be recommended to have a "calibrated_parameters" directory in their calibration work-tree and then we can write functions that will "export to calibrate parameters" (copy optimum parameter sets into the this directory) and then user will use this calibrated_parameters directory to seed subsequent steps of their overall process.

Elchin commented 10 months ago

Note that calibrated parameters do not always get accepted and updated. We used to have https://github.com/uaf-arctic-eco-modeling/dvm-dos-tem/blob/calib/mads_calibration/calib_param_update.py script that updates optimal parameters automatically. Most of the time, the user does it manually.