search

NCAR / amwg_dev

Repo to store model sandboxes and cases used for CAM development
9 stars 2 forks source link

New coupled baseline #349

Closed PeterHjortLauritzen closed 1 year ago

PeterHjortLauritzen commented 1 year ago

Description of the run

B1850 coupled simulation.

Atmosphere: same as https://github.com/NCAR/amwg_dev/issues/345 except

clubb_gamma_coef = 0.30->0.35
clubb_gamma_coefb = 0.3 -> 0.35

Why?

  1. Need new baseline.
  2. latest F-case baseline has RESTOM=1.169, SWCF=-47.551 and LWCF=23.98. By increasing clubb_gamma_coef we expect SWCF to be closer to -45W/m^2 and increasing an RESTOM.

Suffix in the casename

Suffix

Namelist modifications

Namelist

Source modifications

SourceMods

Sandbox

Requested sandbox or tag

Contact info

@cecilehannay @adamrher @JulioTBacmeister @PeterHjortLauritzen

Any other relevant information

Enter relevant info

adamrher commented 1 year ago

I think we should use the new land spun-up inic for this run @olyson @wwieder. Note also that the cloud tunings in this new coupled sim are substantially different from that used to generate the off-line land spinup, so the biases may look quite different from the offline generated climo.

olyson commented 1 year ago

I agree. The new initial conditions are here:

/glade/p/cgd/tss/people/oleson/CLM5_restarts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc

cecilehannay commented 1 year ago

Based on offline discussions with @PeterHjortLauritzen, @adamrher, and @cacraigucar:

jedwards4b commented 1 year ago

Is this to be run on derecho? If so let me know when set up and I can do some load balancing.

cecilehannay commented 1 year ago

Yes, this run will be on derecho. It would be wonderful if you can provide a pe layout.

cecilehannay commented 1 year ago

We are getting ready to start a new coupled run with cesm2_3_alpha16b

The preliminary case directory is: 

/glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16b.BLT1850.ne30_t232.033

The setting is as follow:

ATM

@PeterHjortLauritzen, @adamrher and @JulioTBacmeister

we will be matching: #353

LND

@olyson, @wwieder and @slevis-lmwg

I used SourceMods from 

/glade/u/home/oleson/run_hist_1850_files/SPINUP/casefiles_cecile/SourceMods/ctsm5.1.dev120/src.clm

and user_nl_clm

use_init_interp = .true.
fsurdat = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/surfdata_ne30np4.pg3_SSP5-8.5_78pfts_CMIP6_1850-2100_c230227.nc'
flanduse_timeseries = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/landuse.timeseries_ne30np4.pg3_SSP5-8.5_78_CMIP6_1850-2100_c230227.nc'
finidat='/glade/p/cgd/tss/people/oleson/CLM5_restarts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc'

OCN

@gustavo-marques and @klindsay28:

I added a diag_table in:

/glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16b.BLT1850.ne30_t232.033/SourceMods/src.mom/diag_table

Do I need anything else?

ICE

@dabail10:

Should I use a special ice_ic?

pe_layout

@jedwards4b:

Do you want me to run one month so you can provide a pe_layout?

olyson commented 1 year ago

LND: Please remove the SnowHydrologyMod.F90 SourceMod, it's not longer needed. Please remove flanduse_timeseries = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/landuse.timeseries_ne30np4.pg3_SSP5-8.5_78_CMIP6_1850-2100_c230227.nc'. Since this is an 1850 run, it's not needed. I believe you can remove use_init_interp=.true., because the initial file you are using is generated from a coupler history run that used the new surface dataset. But let me know when it fails :)

dabail10 commented 1 year ago

Not sure about sea ice. If this is the new MOM grid, we don't have a sea ice restart. Maybe Gustavo's G case?

gustavo-marques commented 1 year ago

We do have a restart from a G case with the new ocean grid. Please add the following to user_nl_cice.

ice_ic = '/glade/p/cesmdata/cseg/inputdata/cesm2_init/gmom.e23.GJRAv4.TL319_t061_zstar_N65.tx2_3v2.001/0061-01-01/gmom.e23.GJRAv4.TL319_t061_zstar_N65.tx2_3v2.001.cice.r.0061-01-01-00000.nc'

The MOM6 diag_table is outdated. We will give you a new one tomorrow.

jedwards4b commented 1 year ago

@olyson I think that the use_init_interp=.true. is still needed:

dec2305.hsn.de.hpc.ucar.edu 1:  check_dim_size ERROR: mismatch of input dimension        15920 
dec2305.hsn.de.hpc.ucar.edu 1:   with expected value        15962  for variable gridcell
dec2305.hsn.de.hpc.ucar.edu 1: Did you mean to set use_init_interp = .true. in user_nl_clm?
dec2305.hsn.de.hpc.ucar.edu 1: (Setting use_init_interp = .true. is needed when doing a
dec2305.hsn.de.hpc.ucar.edu 1: transient run using an initial conditions file from a non-transient run,
dec2305.hsn.de.hpc.ucar.edu 1: or a non-transient run using an initial conditions file from a transient run,
dec2305.hsn.de.hpc.ucar.edu 1: or when running a resolution or configuration that differs from the initial conditions.)
dec2305.hsn.de.hpc.ucar.edu 1:  ERROR: 
dec2305.hsn.de.hpc.ucar.edu 1:  ERROR in /glade/work/hannay/cesm_tags/cesm2_3_alpha16b/components/clm/src/main/
dec2305.hsn.de.hpc.ucar.edu 1:  ncdio_pio.F90.in at line 409
jedwards4b commented 1 year ago

Also from cam:

FLDLST: CLD_MISR in fincl(116, 1) not found
 FLDLST: FISCCP1_COSP in fincl(117, 1) not found
 FLDLST: CLD_CAL in fincl(118, 1) not found
 FLDLST: CLD_MISR in fincl(119, 1) not found
 FLDLST: CLDTOT_CAL in fincl(120, 1) not found
 FLDLST: CLDHGH_CAL in fincl(121, 1) not found
 FLDLST: CLDMED_CAL in fincl(122, 1) not found
 FLDLST: CLDLOW_CAL in fincl(123, 1) not found
 FLDLST: CLMODIS in fincl(124, 1) not found
 FLDLST: Uzm in fincl(1, 5) only available with FV dycore
 FLDLST: Vzm in fincl(2, 5) only available with FV dycore
 FLDLST: Wzm in fincl(3, 5) only available with FV dycore
 FLDLST: THzm in fincl(4, 5) only available with FV dycore
 FLDLST: VTHzm in fincl(5, 5) only available with FV dycore
 FLDLST: WTHzm in fincl(6, 5) only available with FV dycore
 FLDLST: UVzm in fincl(7, 5) only available with FV dycore
 FLDLST: UWzm in fincl(8, 5) only available with FV dycore
 ERROR: FLDLST: 9 errors found, see log
jedwards4b commented 1 year ago

Adding ./xmlchange --append CAM_CONFIG_OPTS=-cosp solved the problem with cam FLDLST.

jedwards4b commented 1 year ago

I ran this setup in : /glade/derecho/scratch/jedwards/PFS.ne30pg3_t232.BLT1850_v0c.derecho_intel.20230728_065830_p4afpe I think that it's pretty well balanced as is. 


Overall Metrics:                                                                                                                                                                                                 
    Model Cost:            8247.37   pe-hrs/simulated_year                                                                                                                                                         
    Model Throughput:         6.33   simulated_years/day                                                                                                                                                           ```
cecilehannay commented 1 year ago

The variables below require to as COSP to the CAM_CONFIG_OPT

FLDLST: CLD_MISR in fincl(116, 1) not found
 FLDLST: FISCCP1_COSP in fincl(117, 1) not found
 FLDLST: CLD_CAL in fincl(118, 1) not found
 FLDLST: CLD_MISR in fincl(119, 1) not found
 FLDLST: CLDTOT_CAL in fincl(120, 1) not found
 FLDLST: CLDHGH_CAL in fincl(121, 1) not found
 FLDLST: CLDMED_CAL in fincl(122, 1) not found
 FLDLST: CLDLOW_CAL in fincl(123, 1) not found
 FLDLST: CLMODIS in fincl(124, 1) not found

But the other variables are unrelated to COSP. These are the variables for the TEM diags. I agree that this is a bug that we get this warning (I always get this) and it should be fixed. 

 FLDLST: Uzm in fincl(1, 5) only available with FV dycore
 FLDLST: Vzm in fincl(2, 5) only available with FV dycore
 FLDLST: Wzm in fincl(3, 5) only available with FV dycore
 FLDLST: THzm in fincl(4, 5) only available with FV dycore
 FLDLST: VTHzm in fincl(5, 5) only available with FV dycore
 FLDLST: WTHzm in fincl(6, 5) only available with FV dycore
 FLDLST: UVzm in fincl(7, 5) only available with FV dycore
 FLDLST: UWzm in fincl(8, 5) only available with FV dycore
olyson commented 1 year ago

Thanks @jedwards4b . I guess the use_init_interp is needed because of the mesh mask difference. The Fcase that provided the coupler history files to force the land spinup, and the land spinup itself used:

/glade/p/cesmdata/cseg/inputdata/share/meshes/gx1v7_151008_ESMFmesh.nc

while the fully coupled simulation here uses:

/glade/p/cesmdata/inputdata/share/meshes/tx2_3v2_230415_ESMFmesh.nc

When the first is used, it produces 15920 land gridcells. When the second is used, it produces 15962 gridcells.

cecilehannay commented 1 year ago

@gustavo-marques and @klindsay28

I see that out of the box I get: 

MEKE_GEOMETRIC_ALPHA = 0.07000000000000001

In previous runs, we set:

MEKE_GEOMETRIC_ALPHA = 0.10

Which one do I need?

gustavo-marques commented 1 year ago

Please use MEKE_GEOMETRIC_ALPHA = 0.10

adamrher commented 1 year ago

@olyson are these mesh files the "maskfile" variable? I guess I didn't realize until now that we probably need to change our grid aliases used in the F compsets to use the MOM6 mask for the coastlines.

olyson commented 1 year ago

I looked at the mesh_mask variable in nuopc.runconfig, e.g.,

mesh_mask = /glade/p/cesmdata/cseg/inputdata/share/meshes/gx1v7_151008_ESMFmesh.nc

It looks like that is what it is using to create landfrac and landmask, e.g., from the land log:

Computing land fraction and land mask by mapping mask from mesh_mask file

lnd_set_decomp_and_domain: writing landmask and landfrac data to landfrac.nc

Successfully wrote land fraction/mask status file ./init_generated_files/ctsm_landfrac.status decomp precompute numg,nclumps,seglen1,avg_seglen,nsegspc= 15920 1800 T 1.000000 8.844444 Surface Grid Characteristics longitude points = 48600 latitude points = 1 total number of land gridcells = 15920 Decomposition Characteristics clumps per process = 1

That didn't occur to me either.

dabail10 commented 1 year ago

Why is the land/ocean mask gx1v7? Should it not be the 2/3 degree tripole grid?

adamrher commented 1 year ago

Why is the land/ocean mask gx1v7? Should it not be the 2/3 degree tripole grid?

This just refers to the CLM spinup, not the coupled run. As I said, we didn't add a grid alias for the F-cases that uses the 2/3deg mask. We should probably open an issue. Do we even have the actual new 2/3deg grid in the latest ccs_config?

gustavo-marques commented 1 year ago

Yes, the new 2/3 deg grid was added to ccs_config via https://github.com/ESMCI/ccs_config_cesm/pull/105

adamrher commented 1 year ago

OK thx. Opened an issue: https://github.com/ESMCI/ccs_config_cesm/issues/112

gustavo-marques commented 1 year ago

@cecilehannay, here is the updated diag_table for MOM:

/glade/scratch/gmarques/for_cecile/diag_table

cecilehannay commented 1 year ago

I have still having some issues to start the coupled run: /glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16b.BLT1850.ne30_t232.033 Hopefully, this can be solved before the week-end.

In the mean time, I included the change you suggested above.

klindsay28 commented 1 year ago

@cecilehannay, please try the following: copy $CASEROOT/CaseDocs/input.nml to $CASEROOT/SourceMods/src.mom edit the copy in SourceMods and add the line

    max_num_axis_sets = 26

to diag_manager_nml. This suggestion is based on the last line of ocn.log:

FATAL from PE     0: diag_axis_mod::diag_axis_init: num_axis_sets (26) exceeds max_num_axis_sets (25).  Increase max_num_axis_sets via diag_manager_nml.

This is an FMS namelist, so it is not something you would set from user_nl_mom.

cecilehannay commented 1 year ago

Thanks. I added:

  max_num_axis_sets = 26

and submitted

Right now, the queues on derecho seem to be hanging. But hopefully, the job will start later today.

cecilehannay commented 1 year ago

It set as recommended in the message error @klindsay28 identified:

max_num_axis_sets = 26

But the model is now requesting to set it to 27. 

FATAL from PE     1: diag_axis_mod::diag_axis_init: num_axis_sets (27) exceeds max_num_axis_sets (26).  Increase max_num_axis_sets via diag_manager_nml.

I could keep increasing but I am not sure what this variable does and whether it is an ok strategy.

cecilehannay commented 1 year ago

It is an issue with: 

SourceMods/src.mom/diag_table

as when I remove this mods, it is running fine. (I just tried that).

Should I keep increasing:

max_num_axis_sets
klindsay28 commented 1 year ago

I see that @gustavo-marques has it set to 50 in some of his recent cases, so I suggest going with that.

max_num_axis_sets is an FMS variable used to allocate internal memory for diagnostic axis information. I'm not quite sure why the modified diag_table requires larger values, but it looks like FMS is using it to allocate more space for strings, so it's okay to use larger values. That is, we're not dramatically increasing memory usage by increasing this variable.

cecilehannay commented 1 year ago

Thanks for the explanation. I will set it to 50 (instead of increasing by 1 at a time).

cecilehannay commented 1 year ago

It went further but is now complaining about

max_output_fields =          300

I could increase that too. Any recommendations of a good value?

klindsay28 commented 1 year ago

Could you look at one of Gustavo’s recent cases? I’m away from my computer at the moment.

On Sat, Jul 29, 2023 at 8:40 AM Cecile Hannay @.***> wrote:

It went further but is now complaining about

max_output_fields = 300

I could increase that too. Any recommendations of a good value?

— Reply to this email directly, view it on GitHub https://github.com/NCAR/amwg_dev/issues/349#issuecomment-1656744869, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADWZPO55DPRKYTAHEZKGKL3XSUOFLANCNFSM6AAAAAA2WHAIDA . You are receiving this because you were mentioned.Message ID: @.***>

gustavo-marques commented 1 year ago

@cecilehannay, please copy the following into $CASEROOT/SourceMods/src.mom:

/glade/scratch/gmarques/for_cecile/input.nml

cecilehannay commented 1 year ago

And now it is the max_axes that is exceeded. 

FATAL from PE   169: diag_axis_mod::diag_axis_init: max_axes exceeded, increase via diag_manager_nml

The error message doesn't give suggestion. I set 

max_axes = 50

We'll see.

cecilehannay commented 1 year ago

The model crashed with

max_axes = 50

I set to:

max_axes = 500

It seems to be running now.

klindsay28 commented 1 year ago

Thanks for the update. Sorry for the hassle.

cecilehannay commented 1 year ago

See #356