search

ufs-community / ufs-srweather-app

UFS Short-Range Weather Application
Other
55 stars 116 forks source link

WE2E test "grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16" fails with segmentation fault at run_fcst step #359

Closed mkavulich closed 1 week ago

mkavulich commented 1 year ago

Original issue migrated from regional_workflow repository: https://github.com/ufs-community/regional_workflow/issues/731

Expected behavior

WE2E test "grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16" should succeed.

Current behavior

The failure can manifest in multiple ways: occasionally it gives a segmentation fault with no helpful error messages, other times there is a line FATAL from PE 4: compute_qs: saturation vapor pressure table overflow, nbad= 1 in the output prior to segfault.

Machines affected

This failure has been observed on Hera and Orion, but likely occurs on all platforms.

Steps To Reproduce

  1. On Hera, build the latest UFS_SRW_App and run the grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16 WE2E test
  2. observe failure at the run_fcst step

Output

The full log file for this failing test can be found on Hera at /scratch2/BMC/det/kavulich/workdir/post-merge_testing/expt_dirs/grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16/log/run_fcst_2019052000.log

MichaelLueken commented 3 weeks ago

@mkavulich - Given the removal of the grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16 in PR #732, would it be safe to close this issue?

MichaelLueken commented 1 week ago

Closing issue now. If you feel this issue hasn't been addressed, please feel free to either reopen this issue or open a new issue.