search

ufs-community / regional_workflow

THIS REPOSITORY IS NOW DEPRECATED; SEE UFS SRW APP FOR CURRENT CODE
https://github.com/ufs-community/ufs-srweather-app
Other
10 stars 86 forks source link

WE2E test specify_DOT_OR_USCORE fails for GNU compiler #808

Closed mkavulich closed 2 years ago

mkavulich commented 2 years ago

Description

On Cheyenne for the develop branch, the WE2E test specify_DOT_OR_USCORE fails at the run_fcst step with a segmentation fault:

PASS: fcstRUN phase 1, n_atmsteps =              127 time is        10.738367
PASS: fcstRUN phase 2, n_atmsteps =              127 time is         0.055206
PASS: fcstRUN phase 1, n_atmsteps =              128 time is        10.740046
PASS: fcstRUN phase 2, n_atmsteps =              128 time is         0.061204

FATAL from PE     4: set_group_update: mpp_domains_stack overflow, call mpp_domains_set_stack_size( 5021485) from all PEs.

#0  0x1d2281c in ???
#1  0x219c71e in ???
#2  0x161e877 in __fv_mp_mod_MOD_start_var_group_update_3d
        at /glade/scratch/kavulich/UFS_SRW/testing/test_develop/ufs-srweather-app/src/ufs-weather-model/FV3/atmos_cubed_sphere/tools/fv_mp_mod.F90:794
#3  0x13f4bd8 in __dyn_core_mod_MOD_dyn_core
        at /glade/scratch/kavulich/UFS_SRW/testing/test_develop/ufs-srweather-app/src/ufs-weather-model/FV3/atmos_cubed_sphere/model/dyn_core.F90:442

(see "Output" section for full log file)

This error does not occur with intel compilers, only GNU. I am unable to test GNU compilers for this test on any platform but Cheyenne, so I am not sure if it is a platform-specific problem.

Steps to Reproduce

  1. Build head of develop of ufs-srweather-app with GNU compilers on Cheyenne
  2. Run WE2E test specify_DOT_OR_USCORE
  3. Observe segmentation fault

Additional Context

As mentioned above, I have only observed this failure with GNU compiler on Cheyenne, though it may occur on other platforms.

Output

run_fcst_2019052000.log

mkavulich commented 2 years ago

Issue migrated to ufs-srweather-app repository: https://github.com/ufs-community/ufs-srweather-app/issues/344