search

ufs-community / UFS_UTILS

Utilities for the NCEP models.
Other
21 stars 104 forks source link

regression test failure on hera #612

Closed MinsukJi-NOAA closed 2 years ago

MinsukJi-NOAA commented 2 years ago

Here are the steps I took:

git clone https://github.com/ufs-community/UFS_UTILS
cd UFS_UTILS
./build_all.sh
cd reg_tests/chgres_cube
./driver.hera.sh

In the log files, error messages like this are seen:

 OPEN VERTICAL COORD FILE: 
 /scratch2/NCEPDEV/stmp1/Minsuk.Ji/UFS_UTILS/reg_tests/chgres_cube/../../fix/fix
 _am/global_hyblev.l65.txt
 - FATAL ERROR: OPENING VERTICAL COORD FILE 
 - IOSTAT IS:           29
application called MPI_Abort(MPI_COMM_WORLD, 999) - process 4
MinsukJi-NOAA commented 2 years ago

I was able to run ./link_fixdirs.sh and copy over the files.

GeorgeGayno-NOAA commented 2 years ago

I was able to run ./link_fixdirs.sh and copy over the files.

Most people forget that step. I should automate it somehow.

MinsukJi-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am just checking to see if all RT's pass with ESMF 8.1.0 (as is, without any change). chgres_cube global_cycle grid_gen snow2mdl all passed, but ice_blend failed. Can you please take a look at the file /scratch2/NCEPDEV/stmp1/Minsuk.Ji/UFS_UTILS/reg_tests/ice_blend/out (on Hera)? There seems to be a segfault with /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2

+ /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2 -x -i3 -g '0 0 0 0 0 0 0 0 4320 2160 0 0 89958000 42000 48 -89958000 359958000 83000 83000 0' ims.icec.grib2 ims.icec.5min.grib2
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source    
copygb2            0000000000505EED  Unknown               Unknown  Unknown
libpthread-2.17.s  00007F3CE4409630  Unknown               Unknown  Unknown
copygb2            00000000004076D6  Unknown               Unknown  Unknown
copygb2            000000000040642D  Unknown               Unknown  Unknown
copygb2            0000000000404DAD  Unknown               Unknown  Unknown
copygb2            0000000000403C1E  Unknown               Unknown  Unknown
libc-2.17.so       00007F3CE404E555  __libc_start_main     Unknown  Unknown
copygb2            0000000000403B29  Unknown               Unknown  Unknown
GeorgeGayno-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am just checking to see if all RT's pass with ESMF 8.1.0 (as is, without any change). chgres_cube global_cycle grid_gen snow2mdl all passed, but ice_blend failed. Can you please take a look at the file /scratch2/NCEPDEV/stmp1/Minsuk.Ji/UFS_UTILS/reg_tests/ice_blend/out (on Hera)? There seems to be a segfault with /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2

+ /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2 -x -i3 -g '0 0 0 0 0 0 0 0 4320 2160 0 0 89958000 42000 48 -89958000 359958000 83000 83000 0' ims.icec.grib2 ims.icec.5min.grib2
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source    
copygb2            0000000000505EED  Unknown               Unknown  Unknown
libpthread-2.17.s  00007F3CE4409630  Unknown               Unknown  Unknown
copygb2            00000000004076D6  Unknown               Unknown  Unknown
copygb2            000000000040642D  Unknown               Unknown  Unknown
copygb2            0000000000404DAD  Unknown               Unknown  Unknown
copygb2            0000000000403C1E  Unknown               Unknown  Unknown
libc-2.17.so       00007F3CE404E555  __libc_start_main     Unknown  Unknown
copygb2            0000000000403B29  Unknown               Unknown  Unknown

It is working for me. How did you invoke the script?

ESMF is not used by that program. So, you don't really need to check it. ESMF is only used by chgres_cube and grid_gen.

MinsukJi-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am just checking to see if all RT's pass with ESMF 8.1.0 (as is, without any change). chgres_cube global_cycle grid_gen snow2mdl all passed, but ice_blend failed. Can you please take a look at the file /scratch2/NCEPDEV/stmp1/Minsuk.Ji/UFS_UTILS/reg_tests/ice_blend/out (on Hera)? There seems to be a segfault with /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2

+ /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2 -x -i3 -g '0 0 0 0 0 0 0 0 4320 2160 0 0 89958000 42000 48 -89958000 359958000 83000 83000 0' ims.icec.grib2 ims.icec.5min.grib2
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source    
copygb2            0000000000505EED  Unknown               Unknown  Unknown
libpthread-2.17.s  00007F3CE4409630  Unknown               Unknown  Unknown
copygb2            00000000004076D6  Unknown               Unknown  Unknown
copygb2            000000000040642D  Unknown               Unknown  Unknown
copygb2            0000000000404DAD  Unknown               Unknown  Unknown
copygb2            0000000000403C1E  Unknown               Unknown  Unknown
libc-2.17.so       00007F3CE404E555  __libc_start_main     Unknown  Unknown
copygb2            0000000000403B29  Unknown               Unknown  Unknown

It is working for me. How did you invoke the script?

ESMF is not used by that program. So, you don't really need to check it. ESMF is only used by chgres_cube and grid_gen.

Good to know that I only need to test chgres_cube and grid_gen! For ice_blend, I invoked the same way I did for other utilities: ./driver.hera.sh

GeorgeGayno-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am just checking to see if all RT's pass with ESMF 8.1.0 (as is, without any change). chgres_cube global_cycle grid_gen snow2mdl all passed, but ice_blend failed. Can you please take a look at the file /scratch2/NCEPDEV/stmp1/Minsuk.Ji/UFS_UTILS/reg_tests/ice_blend/out (on Hera)? There seems to be a segfault with /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2

+ /scratch2/NCEPDEV/nwprod/NCEPLIBS/utils/grib_util.v1.1.1/exec/copygb2 -x -i3 -g '0 0 0 0 0 0 0 0 4320 2160 0 0 89958000 42000 48 -89958000 359958000 83000 83000 0' ims.icec.grib2 ims.icec.5min.grib2
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source    
copygb2            0000000000505EED  Unknown               Unknown  Unknown
libpthread-2.17.s  00007F3CE4409630  Unknown               Unknown  Unknown
copygb2            00000000004076D6  Unknown               Unknown  Unknown
copygb2            000000000040642D  Unknown               Unknown  Unknown
copygb2            0000000000404DAD  Unknown               Unknown  Unknown
copygb2            0000000000403C1E  Unknown               Unknown  Unknown
libc-2.17.so       00007F3CE404E555  __libc_start_main     Unknown  Unknown
copygb2            0000000000403B29  Unknown               Unknown  Unknown

It is working for me. How did you invoke the script? ESMF is not used by that program. So, you don't really need to check it. ESMF is only used by chgres_cube and grid_gen.

Good to know that I only need to test chgres_cube and grid_gen! For ice_blend, I invoked the same way I did for other utilities: ./driver.hera.sh

That script must be invoked as: sbatch driver.hera.sh