b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009

[cecilehannay]

Description: New coupled simulation including

• corrected sandbox for cam (revice bugfix)
• new topo file:

  /glade/p/cgd/amp/juliob/bndtopo/latest/ne30pg3_gmted2010_modis_bedmachine_nc3000_Laplace0100_20220622.nc

• bugfix for salinity conservation issue and

  saltflux_option = 'prognostic'

Case directory: Locally (if still available): /glade/p/cesmdata/cseg/runs/cesm2_0/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009

On github: https://github.com/NCAR/amwg_dev/tree/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009

Sandbox: Locally (if still available): /glade/work/hannay/cesm_tags/cesm3_cam6_3_058_MOM_c

On github: https://github.com/ESCOMP/CESM/tree/cesm3_cam6_3_058_MOM hash: 228563a

Diagnostics: AMWG diags (if available) https://webext.cgd.ucar.edu/BWsc1850MOM/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009/atm/

Contacts: @cecilehannay and @gustavo-marques

[cecilehannay]

@gustavo-marques: I am getting an error with the saltflux_option

/glade/work/hannay/cesm_tags/cesm3_cam6_3_058_MOM_c/components/cice/src/cicecore/cicedynB/analysis/ice_diagnostics.F90(224): error #6627: This is an actual argument keyword name, and not a dummy argument name.   [SALTFLUX_OPTION_OUT]
           snwgrain_out=snwgrain, saltflux_option_out=saltflux_option)
----------------------------------^
compilation aborted for /glade/work/hannay/cesm_tags/cesm3_cam6_3_058_MOM_c/components/cice/src/cicecore/cicedynB/analysis/ice_diagnostics.F90 (code 1)
gmake: *** [ice_diagnostics.o] Error 1

[jedwards4b]

I don't see the saltflux_option in icepack_parameters.F90

[cecilehannay]

dabail10updated the cice externals and it solved the issue. This run uses the hash: 228563a

The run is now 9 years.

[gustavo-marques]

So far (year 10) the ocean state looks reasonable. Deep convection is still occurring in the Labrador Sea. The plot below shows the mixed layer depth (MLD) for February of year 9, with the inset showing the MLD time series taken from a point near the southern tip of Greenland.

[gustavo-marques]

The simulation has completed 12 years and it looks like convection in the Labrador Sea is getting weaker :worried:

[JulioTBacmeister]

OK. I still say let's let it go another 8 years. Maybe it will recover.

[cecilehannay]

I am extending the run to 20 years and @justin-richling is processing the ADF. He will let you know when it is ready

[justin-richling]

The ADF diags for the CMIP6 case are here.

There are issues with the Taylor Diagram at the moment, so those and AMWG tables are missing. I am working on getting those fixed and will update soon, but the rest should be good.

The comparison with Obs is currently running and I will update when those are finished

[justin-richling]

ADF diags for comparison vs Obs are here

[cecilehannay]

It looks there is a problem with the SSTs value on the table:
Please ignore the SST value on that table. We will look into that.

[JulioTBacmeister]

Also looks like the SSTs in the latlon plots may also be "weird"

[justin-richling]

Taylor diagrams are now fixed for the CMIP6 case

[cecilehannay]

The run is now 20 years. We are aware that the convection in the Labrador Sea is getting weaker. But @JulioTBacmeister and the cam folks would be VERY interested in letting it go another 20 years. There are clearly some big shocks in the first decade and we need more years to assess this clearly.

@gustavo-marques: what do you think?

[gustavo-marques]

Yes, let's run it for another 20 years. Convection is still happening in the Irminger and Nordic Seas. The Lab Sea is entirely covered by sea ice in the winter, but the ice melts during summer.

[JulioTBacmeister]

Great! Maybe Lab Sea ice cover in winter will begin to decrease ...

[cecilehannay]

We have 55 years now. It would be good to run the diags in the morning.

[JulioTBacmeister]

Quick and dirty IDL diagnostics (below): global TS keeps falling despite global RESTOM averaging around 0.20 to 0.3 Wm-2

Appears to high-latitude cooling that is driving the global mean. A map of mean TS from years 45-54 compared to years 30-45:

[cecilehannay]

This is quite a drop in global TS. Do you know whether it is happening over ocean (or also land)? I wonder if we have a redistribution of the heat in the ocean. It would be nice if @gustavo-marques could look of contours of temperature with time/depth.

On Jul 4, 2022, at 8:57 PM, JulioTBacmeister @.***> wrote:

 Quick and dirty IDL diagnostics (below): global TS keeps falling despite global RESTOM averaging around 0.20 to 0.3 Wm-2

Appears to high-latitude cooling that is driving the global mean. A map of mean TS from years 45-54 compared to years 30-45:

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[cecilehannay]

Indeed your second plot answers to my question about the location of the TS cooling. (sorry I missed the second plot in my email).
This is the high latitude TS.

[swrneale]

@JulioTBacmeister could you tighten your contour interval on your plots as I suspect the ocean could also be cooling at 0-0.5K most places?

[adamrher]

this may be a minor point I think, but we would ideally be running with the exchange grid, as we are currently computing fluxes on the MOM grid. In nuopc.runconfig

     aoflux_grid = ogrid

[justin-richling]

The ADF diags for the years 40-59 vs CMIP6 case are here, comparison vs Obs will be up soon

[JulioTBacmeister]

@swrneale Contours tightened. Now comparing Yrs 45-57 with 30-45. Clearly cooling isn't restricted to high latitudes but is more intense there. Will have a look at regional timeseries.

[cecilehannay]

From looking at ICEFRAC, the lab sea might be still ok. (?) https://webext.cgd.ucar.edu/BWsc1850MOM/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009/atm/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009_40_59_vs_b.e21.B1850.f09_g17.CMIP6-piControl.001_40_59/html_img/plot_page_ICEFRAC_ANN_NHPolar.html

On Jul 5, 2022, at 9:54 AM, justin-richling @.***> wrote:

 The ADF diags for the years 40-59 vs CMIP6 case are here, comparison vs Obs will be up soon

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[gustavo-marques]

Here are global depth versus time temperature biases (model - world ocean atlas). Surface ocean is cooling. Mid-depth ocean is warming a little.

Below is the coupler heat budget (all time up to year 58). It's shown that the ocean is cooling, which is consistent with the plot above. Fluxes are into the atmosphere from all components. Land and ice fluxes to the ATM are both larger than the ocean.

[JulioTBacmeister]

Time series of surface temperature TS in different ocean basins. Definitions are fairly standard. Main point is that tropical oceans (30S-30S) are not cooling, may even be warming after Year 31.

[justin-richling]

ADF diags for years 40-59 vs Obs are here

Misisng AMWG table for b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009 has been added to the CMPI6 case as well, see here

[JulioTBacmeister]

Apologies inadvertently closed comment/issue. Not sure which. @gustavo-marques it would be interesting to see your temperature curtain plots in different basins as well. Is that possible?

[gustavo-marques]

@JulioTBacmeister: yes, here are plots for a few basins.

Lab Sea

Southern Ocean

Indian Ocean

Atlantic Ocean

Pacific Ocean

Arctic

[cecilehannay]

@gustavo-marques: your plot has some similarity with what we observed when we introduced the SE dycore in CESM1. Would you be able to do the same plot (global ocean T timeseries) for the CESM3-MOM6 with the FV dycore? I am just curious to see if it looks similar.

This is what we had with CESM1:

[gustavo-marques]

@cecilehannay:

[cecilehannay]

Thanks @gustavo-marques. The spinup is clearly different between the FV and SE dycores - once again similar to what we observed in CESM1.

Is the ocean initialization similar in SE and FV?

[gustavo-marques]

Yes, ocean initialization is identical in the SE and FV simulations. The ocean starts from rest and T&S are from the January climatology from the World Ocean Atlas 2018.

[adamrher]

Just commenting that I had previously traced that SE warm tongue ~750m to the Mediterranean outflow into the N.Atlantic. Seems to be large changes in SE there... see here a SE-POP run I did a while back.

[gustavo-marques]

this may be a minor point I think, but we would ideally be running with the exchange grid, as we are currently computing fluxes on the MOM grid. In nuopc.runconfig

     aoflux_grid = ogrid

@adamrher: do you know the namelist parameter to be running with the exchange grid?

[jedwards4b]

aoflux_grid = xgrid

[JulioTBacmeister]

So the idea here is that the SE coupled runs have two independent problems/differences from the FV 1) Warm layer at ~750m due to Mediterranean outflow 2) Surface layer/upper ocean cooling due to ???? .... but I would claim looks driven in high latitudes esp northern high latititudes?

Just commenting that I had previously traced that SE warm tongue ~750m to the Mediterranean outflow into the N.Atlantic. Seems to be large changes in SE there... see here a SE-POP run I did a while back.

[adamrher]

@JulioTBacmeister I'd like to know re (2), whether this is driven by local change in the column (clouds?), or changes to ocean currents. As Frank said last week, we'd expect the reduced deep convection in Labrador Sea to cool the the N.Atlantic by slowing down northward heat transport.

Did we run out FV 50 years to be sure it does not cool like SE does? If it does not, then diff'ing the surface energy budget terms should tell us whether the cooling is due to changes in the column from CAM. Here's the first ten years against each other between f09 and this run, from @justin-richling. For example, incident short wave at the surface, which does seem to show ~10 W/m2 reductions in large pockets in the high latitudes.

[gustavo-marques]

I tried to run this case with aoflux_grid = "xgrid" but the model stopped right after initialization due to the following error:

RUNDIR: /glade/scratch/gmarques/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009.xgrid/run/ From PET2209.ESMF_LogFile

20220706 202017.479 ERROR            PET2209 Destination id=280909 NOT found in weight matrix.
20220706 202017.480 ERROR            PET2209 ESMCI_Mesh_Regrid_Glue.C:315 ESMCI_regrid_create() Arguments are incompatible  - - There exist destination cells (e.g. id=280909) which don't overlap with any source cell
20220706 202017.480 ERROR            PET2209 ESMCI_Mesh_Regrid_Glue.C:558 ESMCI_regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20220706 202017.480 ERROR            PET2209 ESMCI_MeshCap.C:1599 MeshCap::regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20220706 202017.480 ERROR            PET2209 ESMF_Regrid.F90:372 ESMF_RegridStore Arguments are incompatible  - Internal subroutine call returned Error
20220706 202017.480 ERROR            PET2209 ESMF_FieldRegrid.F90:1204 ESMF_FieldRegridStoreNX Arguments are incompatible  - Internal subroutine call returned Error
20220706 202017.480 ERROR            PET2209 ESMF_FieldRegrid.F90:3931 ESMF_FieldRegridStoreX Arguments are incompatible  - Internal subroutine call returned Error

@jedwards4b, @adamrher: any ideas on what's the issue here?

[adamrher]

I'm not familiar with the xgrid code. @mvertens? My only suspicion is that many ESMF mesh files are generated from existing SCRIP files. In the past I've ran into issues because some of our SCRIP files are not as precise as they could be.

[jedwards4b]

Can you try with this version of ESMF and let me know if it works: /glade/work/jedwards/tools/esmf/asyncio/lib/libO/Linux.intel.64.mpt.default/esmf.mk

To do this:

cd  /glade/work/hannay/cesm_tags/cesm3_cam6_3_058_MOM_c/ccs_config
git remote add jpedev https://github.com/jedwards4b/ccs_config_cesm.git
git fetch jpedev
git checkout jedwards/cheyenne_esmf_update

Then you need to do

case.setup --reset
case.build

[mvertens]

@gustavo-marques - thanks for trying this. I'm really perplexed that this is not working. I'll try to see what is going on.

[mvertens]

@adamrher - thanks for pointing me to the posting.

[gustavo-marques]

@dabail10, can you please run the ice diagnostics on this case?

[jedwards4b]

@gustavo-marques @mvertens I found the same problem using xgrid with the newer ESMF version.

[mvertens]

@jedwards4b - can you please send a pointer to your rundir.

[gustavo-marques]

Thanks, @jedwards4b. I followed your instructions (updated ESMF) and it's now running: /glade/scratch/gmarques/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009.ogrid2/run It just completed 9 months.

[jedwards4b]

My case is /glade/scratch/jedwards/b.cesm3_cam058_mom_c.B1850WscMOM.ne30_L58_t061.009.xgrid2 I'll compare to @gustavo-marques and see if I can spot the difference.

[jedwards4b]

@gustavo-marques I thought that the problem was with aoflux_grid=xgrid. Your new case is using aoflux_grid=ogrid.

[gustavo-marques]

@jedwards4b: my bad! I cloned the wrong case. Thanks for catching that.