update scorpio submodule to scorpio-v1.4.1

[mahf708]

Could we update scorpio so that we can output IO stats? Thanks.

[mahf708]

@sarats here's updating scorpio, thanks. Not sure who to ping to merge this.

[sarats]

I'm not sure who the maintainers/integrators for this fork are.

Perhaps @wlin7?

[wlin7]

Yes, @sarat. I manage this fork. Is this going to fix the issue #46?

Updating submodule in this way is new to me. Can you or @mahf708 show me how to proceed?

My understanding is users can fetch and reset their scorpio repo to the head of the current scorpio master if to use the updated scorpio. Has it been tested that NGD_v3atm is fully compatible with the updated scorpio?

It has been a puzzle to me how a repo determines the hash of an external submodule to use. In .gitmodule, it only specifies the url and branch (say master of scorpio). Apparently, v3atm is using a pretty old scorpio master. Is the hash to use determined by some kind of inherent link (such as in terms of time stamp) ? I would appreciate if you can point me to some related study material. I wish to know better about it. Until now, I haven't had to deal with such situation. Git submodule update has been working well for me, always fetching and checking the proper hash or branch of the required external module. Thanks.

[sarats]

I don't know if the Scorpio update fixes issue #46. We were talking about I/O performance statistics and that capability is only present in later versions than the one included in this branch. v3atm testing is up to you folks, so please check and then integrate if you have concerns.

As I understand, the submodule hash is stored internally. Roughly speaking, you checkout a specific commit of the submodule, add and then commit that in your branch. General instructions: https://devconnected.com/how-to-add-and-update-git-submodules/#Fetch_new_submodule_commits

[wlin7]

Thanks, @sarats. Your short summary makes perfect sense to me. In that sequence of commands, what committed to the branch must be just a pointer to the specific commit of the submodule. I am going to check out that instruction.

[mahf708]

One way to see what is happening here is to look at this PR as a patch:

From 847c6436cb6d860b0c9bd25bcf839ea0bddd4bee Mon Sep 17 00:00:00 2001
From: mahf708 <naser.mahfouz@pnnl.gov>
Date: Tue, 21 Feb 2023 14:47:33 -0600
Subject: [PATCH] update scorpio

---
 externals/scorpio | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/externals/scorpio b/externals/scorpio
index 1fad001f88..e9618b005f 160000
--- a/externals/scorpio
+++ b/externals/scorpio
@@ -1 +1 @@
-Subproject commit 1fad001f88eba49aac2d1838869515b6eead0691
+Subproject commit e9618b005f91c8d469eae4749ea162da11aa07a9

So my single change is moving the scorpio commit 1fad001..e9618b0. This can also be seen if you look at the GitHub UI:

• https://github.com/E3SM-Project/v3atm/tree/NGD_v3atm/externals
• https://github.com/mahf708/v3atm/tree/NGD_v3atm/externals

This commit movement changed 631 files in the scorpio submodule by bringing it up to date to a tagged version (Dec 2022) as well as v3atm master.

I think your questions underline the tricky nature of dealing with submodules. Not only do they take a long time to get updated (--recusive, etc.) but also they often fall through the cracks in projects' maintenance because they are not obvious targets for upgrades. Obviously they have their benefits, but they do become a hassle in my opinion. Consider a serious situation where one submodules has a severe vulnerability. (Also, recall how some submodules were accidentally partially updated and we had to fix them in https://github.com/E3SM-Project/v3atm/pull/45.) In an ideal world, we would be relying on tagged versions and streamlined upgrades of dependencies (though that tends to slow down some developers).

Anyway, when users update the modules (e.g., before running the model), they only really update them in the sense of requesting recursively the commit stored in the hash on the branch. They do not get anything beyond that. So, this PR takes that commit forward in time. The .submodule file doesn't usually take a specific commit, but maybe it does. I am not sure.

As for testing, I have no idea. I doubt this is breaking anything, and I am actually running two simulations based on this PR on chrysalis right now. However, I am not familiar with the suite of tests needed, especially that no tests are transparent publicly (i.e., in the status checks of PRs here).

[mahf708]

Superseded by #51.

[wlin7]

Thank you much, @mahf708 . Very helpful info on handling submodules. Since you are able to run simulations with this branch, there should be no back compatibility issue. I will do some tests anyway.

FYI: testing and the status for the v3atm are documented on this page.

[mahf708]

I would do the testing on the other PR btw, #51. May even be quicker for you :D

[wlin7]

Certainly. The tests will be done on that PR. Also thanks for bringing that commit to fix the slow build. micro_mg_cam.F90 is not even invoked during runtime for v3atm.