Call MDTF from ADF Revised 20240612

[bitterbark]

• Call MDTF from ADF

This adds a section in the config and run files so the ADF can directly call the MDTF (Model Diagnostics Task Force (MDTF)-Diagnostics package) .

This is the simplest possible implementation that calls an existing, installed version of the MDTF using existing, installed conda envs. For now, timeseries files will be copied to the requested MDTF directory/filenames. Imminent data catalogue functionality in MDTF might change this.

Basic documentation of new functionality: in config.yaml, there is a new section:

diag_mdtf_info: mdtf_run: false [default]

If mdtf_run:true: in run_adf_diag

Call the MDTF:

if diag.get_mdtf_info('mdtf_run'): diag.setup_run_mdtf()

[Detailed documentation of implementation/call tree] (https://www2.cgd.ucar.edu/cms/bundy/Projects/diagnostics/mdtf/notes/run_mdtf_from_adf.html#Documentation)

• Allow hist_str to be a list, and to be different for each cam case (and baseline) The config.yaml file variable can now be a list, and is specified within the diag_cam_climo and diag_cam_baseline_climo lists

  hist_str: [cam.h2,cam.h0]

These timeseries files are all placed in the same ts/ directory, which may need to be modified for multiple component hist strings. (The MDTF checks file types for time frequency for its own purposes).

[bitterbark]

I would like to request a pass on the linter errors. I've fixed all the ones that came from my code (that I can tell; please correct me if I'm wrong). In between my previous attempt at a PR and this time, it seems that others' code has been committed successfully without resolving any of these issues. The problems are not in my code, and the work I put in last time to clean up the code just made it even more difficult to merge in the branch this time (as I essentially had to take the cleanup out to be compatible with the current main branch), not to mention adding more merging work when other PRs came in before mine.

Can we please get this PR in and then do the (unrelated) code cleanup separately so we can all start from a clean slate?

[nusbaume]

@bitterbark I have gone ahead and fixed the linter errors/warnings. I'll try to give this PR one last look over, and if OK on my end will send it over to @justin-richling to run a test or two. Thanks!

[bitterbark]

Thanks, Jesse (and Justin). I’m on vacation until June 25 so I can work then on anything that comes up in the testing.On Jun 13, 2024, at 13:47, Jesse Nusbaumer @.***> wrote: @bitterbark I have gone ahead and fixed the linter errors/warnings. I'll try to give this PR one last look over, and if OK on my end will send it over to @justin-richling to run a test or two. Thanks!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[bitterbark]

I took the merged code from Justin but am running into errors when I try to push it to my branch. First question: it looks like the changes Justin made might already be committed to my branch. Is there a chance I don't need to make any other mods other than the one suggestion? Unfortunately this thread shows extensive merge conflicts that i don't see how to resolve.

So when I try to push the new files from Justin up to my branch, I get merge errors. Honestly not different from what happened earlier this month, when it said my online branch was ahead of the local branch (somehow) and then I let it do the merge and it messed everything up and I had to redo all my changes by hand.

Wondering if one of you has advice.

git push ! [rejected] call_mdtf_from_adf.20240612 -> call_mdtf_from_adf.20240612 (fetch first) error: failed to push some refs to 'github.com:bitterbark/ADF.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.

So then I try

git fetch ... Unpacking objects: 100% (47/47), 37.01 KiB | 2.00 KiB/s, done. From github.com:bitterbark/ADF 63c2c90..3c541f2 call_mdtf_from_adf.20240612 -> origin/call_mdtf_from_adf.20240612

and

git pull hint: You have divergent branches and need to specify how to reconcile them. hint: You can do so by running one of the following commands sometime before hint: your next pull: hint: hint: git config pull.rebase false # merge hint: git config pull.rebase true # rebase hint: git config pull.ff only # fast-forward only

I really don't want to have to merge by hand again. Surely there is some way git can do this. Advice?

[nusbaume]

@bitterbark @justin-richling I have resolved the merge conflicts and gotten the linting tests to pass again. The merge was a little dicey so please retest this branch with your relevant ADF workflow whenever you get the chance. Assuming everything works for you then I think (?) this PR is good to go. Thanks!

[bitterbark]

Thanks so much Jesse!