NOAA-EMC / GDASApp

Global Data Assimilation System Application
GNU Lesser General Public License v2.1
15 stars 31 forks source link

gdas_aeroanlgenb fails in g-w CI C96C48_hybatmaerosnowDA #1381

Closed RussTreadon-NOAA closed 3 days ago

RussTreadon-NOAA commented 3 days ago

g-w CI C96C48_hybatmaerosnowDA job gdas_aeroanlgenb failed for 20211220 12Z with the error message

found unhashable key
  in "<unicode string>", line 88, column 11:
        nxp: {{ npx_rescale }}
              ^

This error is due to two factors

  1. Typo: the key nxp should be npx in parm/aero/berror/aero_diagb.yaml.j2
  2. Missing setting. Neither npx_rescale nor npy_rescale are set

A check of g-w parm/config/gfs/config.aeroanlgenb finds

export aero_diffusion_fixed_val=1.0
export npx_clim_b=97
export npy_clim_b=97
export aero_diagb_weight=0.9

A check of GDASApp parm/aero/jcb-base.yaml.j2 finds

aero_npx_anl: {{ npx_anl | default(49, true) }}
aero_npy_anl: {{ npy_anl | default(49, true) }}
aero_npz_anl: {{ npz_anl | default(127, true) }}
aero_npx_clim_b: {{ npx_clim_b | default(49, true) }}
aero_npy_clim_b: {{ npy_clim_b | default(49, true) }}
aero_npz_clim_b: {{ npz_anl | default(127, true) }}

aero_fv3jedi_files_path: ./fv3jedi  # Ideally this would be {{DATA}}/fv3jedi but FMS

Neither of these files set or provide defaults for npx_rescale and npy_rescale.

Correcting item 1 is easy. It's not clear to me what is the correct way to fix 2. Should npx_rescale and npy_rescale be added to both config.aeroanlgenb and parm/aero/jcb-base.yaml.j2

A test in which npx_rescale and npy_rescale was added to both files was not sufficient for gdas_aeroanlgenb to successfully run to completion. Executable gdasapp_chem_diagb.x aborted with

 0:  cubic_grid domain decomposition
 0: whalo =    3, ehalo =    3, shalo =    3, nhalo =    3
 0:   X-AXIS =   24  24  24  24
 0:   Y-AXIS =   24  24  24  24
33: fv3jedi_io_fms_mod.read_restart_fields: file .//
64: fv3jedi_io_fms_mod.read_restart_fields: file .//
36: fv3jedi_io_fms_mod.read_restart_fields: file .//
39: fv3jedi_io_fms_mod.read_restart_fields: file .//
71: fv3jedi_io_fms_mod.read_restart_fields: file .//
91: fv3jedi_io_fms_mod.read_restart_fields: file .//
64: Abort(1) on node 64 (rank 64 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 64
47: fv3jedi_io_fms_mod.read_restart_fields: file .//
71: Abort(1) on node 71 (rank 71 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 71
52: fv3jedi_io_fms_mod.read_restart_fields: file .//

Open this issue to report this problem and document its resolution.

RussTreadon-NOAA commented 3 days ago

Until this issue is resolved g-w PR #3118 can not move forward.

RussTreadon-NOAA commented 3 days ago

Tagging @andytangborn and @CoryMartin-NOAA for awareness. I'm guessing the failure is related to GDASApp PR #1339. Are additional changes needed to GDASApp, jcb-gdas, and/or g-w?