Update regrid.pl to include MOM6 ocean grids and new location for MOM5 grids.

[yvikhlya]

Currently, regrid.pl looks for coupled tile files in:

$coupled_model_dir = "/discover/nobackup/yvikhlia/coupled/Forcings";

and supports only following MOM5 configurations:

$imo{"cc"} =  "360"; $jmo{"cc"} = "200";
$imo{"dd"} =  "720"; $jmo{"dd"} = "410";
$imo{"ee"} = "1440"; $jmo{"ee"} = "1080";

Recently, coupled model tile files were moved to /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM5 and /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-Nlv3/MOM6. Two MOM6 configuration were added 72x36 and 1440x1080 (the latter is not the same as MOM5 1440x1080). More MOM6 configurations are possible in the future. We need regrid.pl script to be able to produce restarts for all these configurations.

I am not an expert in perl, but I can start working on regrid.pl and probably need help from @bena-nasa.

[sanAkel]

@bena-nasa, @mathomp4, @atrayano, @tclune, @GEOS-ESM/gmao-si-team, appreciate your help with this!

Specifically with the 5-deg, i.e., C24 (atm) x 72x36 (ocn) case. Thanks a lot!

[yvikhlya]

We need logic, which distinguishes between MOM5 and MOM6. I don't know how to implement this in perl.

[mathomp4]

We might have to do some "ugliness".

1. Make the current cc, dd, and ee oceans: cc5, dd5, and ee5 so that we can add cc6, etc. But we don't have "negative" letters. If you follow the pattern, aa6 would be 90x something. If we don't care that much, then we just say aa6 is 72x36 and live with that
2. Create a new set of mom6 names that don't do the cc etc. pattern. We could do anything you like. mom6-72 and mom6-1440 or whatever.

[mathomp4]

Or I suppose, leave cc, dd, etc. alone and make the MOM6 indicators ee6 etc.

[yvikhlya]

Or I suppose, leave cc, dd, etc. alone and make the MOM6 indicators ee6 etc.

Note: we are not trying to regrid MOM5/6 restarts here, we want to regrid atmosphere and surface restarts. Surface restarts are tile restarts and we need to point regrid.pl to correct tile file, e.g.

/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM5/CF0180x6C_TM1440xTM1080/CF0180x6C_TM1440xTM1080-Pfafstetter.til  - for MOM5

/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0180x6C_TM1440xTM1080/CF0180x6C_TM1440xTM1080-Pfafstetter.til - for MOM6

Regridding of coupled restarts is essentially no different than regridding of AMIP restarts. "Icarus-NLv3" in the above example is a version of land boundary conditions, which has to be a variable. Then there are 3 other variables, ocean model version (MOM5/MOM6, later MIT ocean is possible), atmosphere resolution (c180x6 in the above example) and ocean resolution (1/4 degree in the above example), all this information should point to a correct tile file. The rest of the regridding logic is the same as in AMIP case afaik.

[mathomp4]

@yvikhlya I'm trying to look at cleaning up regrid.pl now. Are you planning on copying over the older coupled tile files to the new location? Because right now, the only things in the new location are Icarus-NLv3. I don't see any of the older files there.

Or are the MOM5 files in the new location the same as the files that were in the old location?

Or should we just no longer support any of the older tile?

I'm just wondering how to handle the new files and if I need to keep support for the older ones.

[yvikhlya]

The idea is to consolidate boundary conditions and include coupled model support. Right now the new location has only Icarus-NLv3 bcs, because I am not sure what to do with old bcs. We can't abandon support of older tile files because S2Sv2/v3 use them, so they should stay where they are right now, but from now on it would be good to handle different versions of bcs some better way than we do now. As for regrid.pl, I would suggest not to worry about old tile files for now, but to add support of new tile files in such a way that bcs version, ocean model (MOM5/MOM6/MIT) and resolution are variables. We need a straightforward way to generate restarts for our S2Sv4 and other experiments. @tclune @amolod please share your opinion.

[mathomp4]

@yvikhlya When you have a chance, can you please make the topography symlinks in these new directories? regrid.pl requires (at least?) the topo_DYN_ave data file to be present to work.

[sanAkel]

Thank you @mathomp4, I am sure @yvikhlya would be able to help with ⤴️ Thanks!

[yvikhlya]

@mathomp4 @sanAkel Done. All is there /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM5/6

[mathomp4]

@yvikhlya Thanks.

I just gave it a try with my test regrid.pl and it seems to not crash. I regridded from MERRA-2 (original flavor) on 20000414 21z to MOM6 C24 72x36 (that is, I used my usual MERRA2 restarts). I suppose @sanAkel can be a tester for us and see if all works? You can find the output here:

/gpfsm/dnb44/mathomp4/Regrid/OceanRegridTry/FromMattMerra2-C24-MOM6-CMDLine

[sanAkel]

Thanks @mathomp4 but I am not sure if MERRA2 restarts are good with new land. @yvikhlya can you please verify what you had to do for the OM4 coupled model you are testing (.e., 0.25deg MOM6 x C180 atmosphere).

Did MERRA-2 restarts work or you used something else?

[yvikhlya]

Oh, that is true, MERRA-2 restarts did not work and I used m2ocean restarts.

[mathomp4]

Thanks @mathomp4 but I am not sure if MERRA2 restarts are good with new land. @yvikhlya can you please verify what you had to do for the OM4 coupled model you are testing (.e., 0.25deg MOM6 x C180 atmosphere).

Did MERRA-2 restarts work or you used something else?

@sanAkel Well, I'm not thinking it'll be good science. I just wondered if they'd run. As in, the model doesn't die after a day.

[sanAkel]

@mathomp4 for sure, the MOM6 C24 72x36 is not for any scientific work. But I think even for running 1day1 @yvikhlya's recent experience with new land and MERRA-2 restarts is relevant. We are both on leave tomorrow 10/15, we will try on Monday/next week and get back to you. 🙏

[mathomp4]

@sanAkel Okay. I think @aoloso is going to look at adding MERRA2-NL and options to get good seaice into the script as well.

[yvikhlya]

@mathomp4 We can't use MERRA2 restarts with coupled Jason model, because seaice_thermo restart is incompatible and it can't be bootstrapped or converted. For S2Sv4 model I took m2ocean restarts from /discover/nobackup/projects/gmao/merra2-ocean/M2OCEAN_S2SV3 and I think that the same restarts will work for c24 model too (I made a copy to /home/yvikhlia/tmp/s2sv4/restarts_m2o). Corresponding source tile file is located at /home/yvikhlia/nobackup/coupled/Forcings/a180x1080_o1440x1080. Destination tile file is at /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0024x6C_TM0072xTM0036 (from now on we will use aogcm disk to host all coupled data files).

My questions are:

1. Which version of regrid.pl should I use?
2. Should I copy tile file (and other data files) from /home/yvikhlia/nobackup/coupled/Forcings/a180x1080_o1440x1080 to /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/...?

@sanAkel Please comment if you see anything wrong.

[sanAkel]

@sanAkel Please comment if you see anything wrong.

@yvikhlya I can't comment on the ⤴️ However, I would like to add following:

• For S2Sv4 model I took m2ocean restarts @mathomp4, S2Sv4 implies MOM6 ocean and GEOS atmosphere, at 1/4 deg and 1/2 deg (C180) respectively, tile file pointed by @yvikhlya makes this clear.

@yvikhlya it would be good to note, for the record why:

because seaice_thermo restart is incompatible and it can't be bootstrapped or converted.

• what is the reason for incompatibility?
• why it cannot be converted?
• I could imagine, why bootstrap isn't a viable!

[mathomp4]

@yvikhlya If you want to test what I have (even if it's not correct), you'll need to use my branch:

feature/mathomp4/update-regrid-pl

It's the only place I've put on my attempt to use your new files

[yvikhlya]

• what is the reason for incompatibility?

• why it cannot be converted?

• I could imagine, why bootstrap isn't a viable!

MERRA2 has old saltwater format. When I tried to regrid MERRA2 to S2Sv4, it split saltwater_internal into openwater_internal and seaicethermo_internal and the latter crashed the model. @zhaobin74 advised not to take seaicethermo_internal from MERRA2.

[mathomp4]

Andrea had mentioned that @zhaobin74 might have a utility that can convert the seaice? Not sure

[mathomp4]

Also, pinging @aoloso on this thread as he might be doing some of the regrid.pl work going forward

[zhaobin74]

It's not a good idea to convert AMIP (i.e. SimpleSeaice) seaicethermo restart to coupled as it lacks critical information, e.g. ice thickness. Coupled model needs to restart from proper sea ice volume such that any drift can be attributed to the model, not the initial condition. We have seen many examples of the bad initialization.

I'd suggest regrid.pl should pick up MERRA-2 OCEAN restarts when it does regridding of seaicethermo restart.

Andrea had mentioned that @zhaobin74 might have a utility that can convert the seaice? Not sure

[zhaobin74]

@sanAkel Please comment if you see anything wrong.

@yvikhlya I can't comment on the ⤴️ However, I would like to add following:

* `For S2Sv4 model I took m2ocean restarts` @mathomp4, `S2Sv4` implies MOM6 ocean and GEOS atmosphere, at 1/4 deg and 1/2 deg (C180) respectively, tile file pointed by @yvikhlya makes this clear.

@yvikhlya it would be good to note, for the record why:

because seaice_thermo restart is incompatible and it can't be bootstrapped or converted.

* what is the reason for incompatibility?

It is AMIP SimpleSeaice restart, not the CICEColumnPhys.

* why it cannot be converted?

It lacks information to properly represent sea ice volume and its spatial distribution.

* I could imagine, why bootstrap isn't a viable!

It can not be bootstrapped just like other tile restarts, Land, Lake, Landice.

[mathomp4]

It's not a good idea to convert AMIP (i.e. SimpleSeaice) seaicethermo restart to coupled as it lacks critical information, e.g. ice thickness. Coupled model needs to restart from proper sea ice volume such that any drift can be attributed to the model, not the initial condition. We have seen many examples of the bad initialization.

I'd suggest regrid.pl should pick up MERRA-2 OCEAN restarts when it does regridding of seaicethermo restart.

Andrea had mentioned that @zhaobin74 might have a utility that can convert the seaice? Not sure

@zhaobin74 Ah. Thanks. I know Andrea mentioned something called convrstSALT and I didn't know what that did but I guess I'd thought it would "fix" a MERRA2 restart. Guess not!

[sanAkel]

thanks @zhaobin74

[zhaobin74]

It's not a good idea to convert AMIP (i.e. SimpleSeaice) seaicethermo restart to coupled as it lacks critical information, e.g. ice thickness. Coupled model needs to restart from proper sea ice volume such that any drift can be attributed to the model, not the initial condition. We have seen many examples of the bad initialization. I'd suggest regrid.pl should pick up MERRA-2 OCEAN restarts when it does regridding of seaicethermo restart.

Andrea had mentioned that @zhaobin74 might have a utility that can convert the seaice? Not sure

@zhaobin74 Ah. Thanks. I know Andrea mentioned something called convrstSALT and I didn't know what that did but I guess I'd thought it would "fix" a MERRA2 restart. Guess not!

@mathomp4, I did write such a converter years ago as a response to a very specific request. It has never been meant for general use.

The bottom line is AMIP/MARRA-2 seaicethermo restart is a bad starting point for coupled, and we have many good ones sitting around.

[yvikhlya]

@mathomp4 I think I found a little bug here https://github.com/GEOS-ESM/GMAO_Shared/blob/8b1df206067cca2d1d641a18180b35ec8958a01f/GEOS_Util/post/regrid.pl#L789 It says "Enter OUTPUT ocean model" while in fact it reads input ocean model.

[mathomp4]

@mathomp4 I think I found a little bug here

https://github.com/GEOS-ESM/GMAO_Shared/blob/8b1df206067cca2d1d641a18180b35ec8958a01f/GEOS_Util/post/regrid.pl#L789

It says "Enter OUTPUT ocean model" while in fact it reads input ocean model.

Sigh. oops. Yeah, the "power" of copy-paste. Let me fix that...

[yvikhlya]

@mathomp4 Also, mdlINocean is uninitialized at this point https://github.com/GEOS-ESM/GMAO_Shared/blob/8b1df206067cca2d1d641a18180b35ec8958a01f/GEOS_Util/post/regrid.pl#L788 which causes a warning printed. The same is with mdlOUTocean here https://github.com/GEOS-ESM/GMAO_Shared/blob/8b1df206067cca2d1d641a18180b35ec8958a01f/GEOS_Util/post/regrid.pl#L814

[mathomp4]

Okay. I think I fixed that.

[mathomp4]

Okay some updates. I've pushed some changes. Now, it is a bit more exciting. There are now some questions about ocean models.

So, I'm going to regrid some S2S restarts I found, a la:

S2S-2_1_ANA_001.saltwater_import_rst.e20210814_00z.Heracles-5_4_p3.a180x1080_o720x410_CF0180x6C_DE0360xPE0180.nc4

to C24 MOM6. As I run regrid it asks:

Ocean Models
------------
data  (Reynolds, MERRA-2, Ostia, Cubed-Sphere)
MOM5
MOM6

Enter INPUT ocean model [data]

Now the default here is data because that's the most "usual". But I'm going to answer MOM5 (because I think that's what S2S is???). Once you do that, it knows your input models is coupled so it will ONLY allow you to go to coupled outputs. (If you say the inputs are data ocean, it will ONLY allow you to regrid to data ocean.)

Enter INPUT ocean model [data] MOM5
INPUT ocean model: MOM5

Coupled Ocean Grids
-------------------
aa = 72x36
cc = 360x200
dd = 720x410
ee = 1440x1080

Enter INPUT ocean grid: [dd]
INPUT ocean grid: dd
Enter OUTPUT ocean model [MOM6]
OUTPUT ocean model: MOM6
Enter OUTPUT ocean grid: [dd] aa
OUTPUT ocean grid: aa

and after that it's as it is.

Now it's still not smart. You have to know that MOM6 C24 for Icarus-NLv3 is allowed, but it's a first start.

But in the end I get restarts as seen here:

/discover/nobackup/mathomp4/Regrid/OceanRegridTry/c24-mom6-from-S2S

and the resulting seaicethermo restart does have 23 variables instead of 13

[mathomp4]

Also, @yvikhlya, if you have a chance, can you copy all the "old" BCs you have in:

/discover/nobackup/yvikhlia/coupled/Forcings

into the new place:

/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs

I see you have Icarus and Icarus-NLv3 in the new location, but it might clean up the code if we have them all in one place.

I suppose it would be good if you could put them in similar BCType/Resolution directory trees?It looks like most of the old BCs are for Ganymed but some seem to have no name on them.

[yvikhlya]

Hi Matt, my understanding is that Icarus and Icarus-NLv3 are BCTypes and this is all we have for coupled configuration. Unless I am missing something. I would like to have clear directions from land group or somebody, where I need to copy coupled data files, because I am somewhat confused now. @mathomp4

[sanAkel]

@mathomp4 let me reach over teams. Thanks!

[mathomp4]

Hi Matt, my understanding is that Icarus and Icarus-NLv3 are BCTypes and this is all we have for coupled configuration. Unless I am missing something. I would like to have clear directions from land group or somebody, where I need to copy coupled data files, because I am somewhat confused now. @mathomp4

Huh Okay. I was hoping to clean/remove these lines involving coupled_model_dir/coupled_model_tile:

https://github.com/GEOS-ESM/GMAO_Shared/blob/c2bbdae426d703b4ee695e7c324435378b9e1f56/GEOS_Util/post/regrid.pl#L106-L125

I don't think they are dead code...but maybe? I could try commenting them out and regridding S2S to MOM6. If nothing is triggered, then it's dead code! :smile:

[yvikhlya]

Hi Matt, my understanding is that Icarus and Icarus-NLv3 are BCTypes and this is all we have for coupled configuration. Unless I am missing something. I would like to have clear directions from land group or somebody, where I need to copy coupled data files, because I am somewhat confused now. @mathomp4

Huh Okay. I was hoping to clean/remove these lines involving coupled_model_dir/coupled_model_tile:

https://github.com/GEOS-ESM/GMAO_Shared/blob/c2bbdae426d703b4ee695e7c324435378b9e1f56/GEOS_Util/post/regrid.pl#L106-L125

I don't think they are dead code...but maybe? I could try commenting them out and regridding S2S to MOM6. If nothing is triggered, then it's dead code! 😄

It is a dead code from my point of view. I would clean this up if I understood perl a little bit better.

[mathomp4]

It is a dead code from my point of view. I would clean this up if I understood perl a little bit better.

@yvikhlya Well, that gives me a good reason to try removing it and seeing what happens! I'll report back soon

[yvikhlya]

It is a dead code from my point of view. I would clean this up if I understood perl a little bit better.

@yvikhlya Well, that gives me a good reason to try removing it and seeing what happens! I'll report back soon

Thanks Matt!

[mathomp4]

@yvikhlya Actually it is not dead code. I spoke a bit with @bena-nasa and the regridding code needs both the input and output tile files and input/output topo files. And since I'm regridding from S2S restarts, we need the Ganymed stuff you have in /discover/nobackup/yvikhlia/coupled/Forcings.

As a test, I whipped up a skeleton bcs area here:

/discover/nobackup/mathomp4/atmosphere_bcs

where I've symlinked your Icarus and Icarus-NLv3 directories as well as created a Ganymed/MOM5 tree:

/discover/nobackup/mathomp4/atmosphere_bcs/Ganymed/
`-- MOM5
    |-- CF0048x6C_TM0360xTM0200
    |-- CF0090x6C_TM0360xTM0200
    |-- CF0090x6C_TM0720xTM0410
    |-- CF0090x6C_TM1440xTM1080
    |-- CF0180x6C_TM0720xTM0410
    |   |-- INPUT
    |   |   `-- dave
    |   |-- cice
    |   |-- clsm
    |   |-- mk_MAPL_Tripolar
    |   `-- save
    |       `-- h30
    `-- CF0180x6C_TM1440xTM1080

In there I copied the files from:

/discover/nobackup/yvikhlia/coupled/Forcings/a180x1080_o720x410/

into my Ganymed/MOM5/CF0180x6C_TM0720xTM0410/ directory as well as linked in the input topography:

/discover/nobackup/projects/gmao/share/gmao_ops/fvInput/g5gcm/bcs/Ganymed-4_0/Ganymed-4_0_Reynolds/CF0180x6C_DE0360xPE0180/topo_DYN_ave_180x1080.data

When I did this, and pointed regrid.pl to my new directory, I could then regrid from S2S to MOM6 as before.

So, if you have time, it might be worth making this tree for Ganymed/MOM5 just like you have for the Icarus and Newland. If you do that, then the coupled code is simplified since there is just one place it looks for things.

(Of course, don't delete the old ones since that would break all other regrid.pl out there.)

[yvikhlya]

Oh, I got it now. Ganymed tile file is actually no different from Icarus tile file but if this is necessary to have different locations, I'll make a copy.

[yvikhlya]

@mathomp4 I did copy your whole tree into /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs. Some of these configurations are dysfunctional and need to be deprecated (I'll take care of this). I support the idea to simplify coupled code, such that we can add/remove configurations without pain.

[mathomp4]

@mathomp4 I did copy your whole tree into /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs. Some of these configurations are dysfunctional and need to be deprecated (I'll take care of this). I support the idea to simplify coupled code, such that we can add/remove configurations without pain.

@yvikhlya Okay, I see that you have the S2S BCs (Ganymed/MOM5/CF0180x6C_TM0720xTM0410) there now. I suppose that's enough for my testing purposes. I'll look at changing my code up to use the one directory space. And then as you fill it out, we'll see which resolutions you are happy with.

[sanAkel]

Reopened to fix c24x5-deg case. @atrayano appreciate your help. Thank you!