Replace 'print' with logging. Prints in the utility function remain. All the loggings are stored in the file run.log.
Create a utility function for ensemble model check. Conditions can be added or changed based on further discussion.
Add an ADR for log files (009), generated file naming convention(004), evaluation metrics(010).
Move utils_wandb.py to common_utils
Add one new ensemble model (aggregated by median)
This is the newest version of orchestration including managing ensemble models. Previous ones are closed but haven't been reviewed. Below are things worth mentioning:
For ensemble model
The code only supports the old stepshifter. Double-check the targets as they should be the same for aggregation (log or non-log, considering adding a check to this but how to get config files elegantly is a problem).
Forecasting directly aggregates the output of each model (if exists). Evaluating uses the artifact of each model and then aggregates the predictions because df_output_dict changes the structure of original outputs and we don't store the original outputs for calibration and testing. In the future, we will have a script that produces the evaluation matrix with the same structure.
For orchestration
The script executes every main.py file in every model and ensemble folder. Currently, it only allows you to do either single models or ensemble models, which is decided by whether the argument '--aggregation' is provided.
Refer to orchestration/README.md to execute the codes.
For dataloader:
There is a mismatch in the format between stepshifter model and hydranet. For stepshifter, the month and the level are indexes but for hydranet they are columns. I just commented out those codes but there will be errors for hydranet models if the branch is merged.
Changes in this pr:
This is the newest version of orchestration including managing ensemble models. Previous ones are closed but haven't been reviewed. Below are things worth mentioning:
For ensemble model
For orchestration The script executes every main.py file in every model and ensemble folder. Currently, it only allows you to do either single models or ensemble models, which is decided by whether the argument '--aggregation' is provided. Refer to orchestration/README.md to execute the codes.
For dataloader: There is a mismatch in the format between stepshifter model and hydranet. For stepshifter, the month and the level are indexes but for hydranet they are columns. I just commented out those codes but there will be errors for hydranet models if the branch is merged.