search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
75 stars 168 forks source link

Cannot successfully build develop on hera #837

Closed JessicaMeixner-NOAA closed 2 years ago

JessicaMeixner-NOAA commented 2 years ago

Expected behavior Should be able to checkout and build on hera.

Current behavior The errors from the build: 

Creating ../exec folder
 .... Building WW3 pre and post execs .... 
 .... Building forecast model .... 
Fatal error in building UFS model.
The log file is in /scratch1/NCEPDEV/climate/Jessica.Meixner/devworkflow/global-workflow/sorc/logs/build_ufs.log
 .... Building gsi .... 
Fatal error in building gsi.
The log file is in /scratch1/NCEPDEV/climate/Jessica.Meixner/devworkflow/global-workflow/sorc/logs/build_gsi.log
 .... Building UPP .... 
Fatal error in building UPP.
The log file is in /scratch1/NCEPDEV/climate/Jessica.Meixner/devworkflow/global-workflow/sorc/logs/build_upp.log
 .... Building ufs_utils .... 
 .... Building gldas .... 
Fatal error in building gldas.
The log file is in /scratch1/NCEPDEV/climate/Jessica.Meixner/devworkflow/global-workflow/sorc/logs/build_gldas.log
 .... Building workflow_utils .... 
Fatal error in building workflow_utils.
The log file is in /scratch1/NCEPDEV/climate/Jessica.Meixner/devworkflow/global-workflow/sorc/logs/build_workflow_utils.log
 .... Building gfs_util .... 
FATAL BUILD ERROR: Please check the log file for detail, ABORT!

Machines affected hera (trying orion now)

To Reproduce On hera: 

git clone https://github.com/noaa-emc/global-workflow
cd global-workflow/
cd sorc/
sh checkout.sh
sh build_all.sh

Context Trying to compare updates in a PR, ran into new issues and noted these are also issues in develop. Wanted to report the issue in case others were experiencing the same problem.

Detailed Description The build fails for UPP, GSI, GLDAS, and workflow utils

JessicaMeixner-NOAA commented 2 years ago

UFS error: 

[100%] Linking Fortran executable ufs_model
/scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a(addgrid.f.o): In function `addgrid_':
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/addgrid.f:92: undefined reference to `for_stop_core_quiet'
/scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a(compack.f.o): In function `compack_':
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:382: undefined reference to `for_stop_core_quiet'
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:351: undefined reference to `for_stop_core_quiet'
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:156: undefined reference to `for_stop_core_quiet'
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:152: undefined reference to `for_stop_core_quiet'
/scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a(compack.f.o):/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:141: more undefined references to `for_stop_core_quiet' follow
make[2]: *** [ufs_model] Error 1
make[1]: *** [CMakeFiles/ufs_model.dir/all] Error 2
make: *** [all] Error 2

GSI error: 

++ /apps/lmod/lmod/libexec/lmod sh load modulefile.ProdGSI.hera
Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "w3nco/2.4.1"
   Try: "module spider w3nco/2.4.1" to see how to load the module(s).

UPP error: 

Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "w3nco/2.4.1"
   Try: "module spider w3nco/2.4.1" to see how to load the module(s).

GLDAS error is unclear

Workflow utils error: 


/scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/nemsio/2.5.4/lib64/libnemsio.a(nemsio_openclose.f90.o): In function `nemsio_openclose_mp_nemsio_stop_':
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/nemsio-v2.5.4/src/nemsio_openclose.f90:5138: undefined reference to `for_stop_core_quiet'
/scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/nemsio/2.5.4/lib64/libnemsio.a(nemsio_openclose.f90.o): In function `nemsio_openclose_mp_nemsio_stop_':
/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/nemsio-v2.5.4/src/nemsio_openclose.f90:5138: undefined reference to `for_stop_core_quiet'
make[2]: *** [fv3nc2nemsio.fd/fv3nc2nemsio.x] Error 1
make[1]: *** [fv3nc2nemsio.fd/CMakeFiles/fv3nc2nemsio.x.dir/all] Error 2
make[2]: *** [regrid_nemsio.fd/regrid_nemsio.x] Error 1
make[1]: *** [regrid_nemsio.fd/CMakeFiles/regrid_nemsio.x.dir/all] Error 2
[ 42%] Built target enkf_chgres_recenter_nc.x
make: *** [all] Error 2

@Hang-Lei-NOAA @kgerheiser have there been module updates in hpc-stack on hera recently that might help explain the above errors? @WalterKolczynski-NOAA any guess on when things last worked in the develop branch of the workflow (time/commit)?

JessicaMeixner-NOAA commented 2 years ago

The issue was I had module use and conflicting modules loaded before doing build_all.sh. A clean login resolved the issues.

KateFriedman-NOAA commented 2 years ago

Good to know what that looks like in case someone else reports something similar! :)

Hang-Lei-NOAA commented 2 years ago

-bash-4.2$ ll /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a -rw-r--r-- 1 Hang.Lei nwprod 934020 Mar 16 01:02 /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a

lib g2/3.4.2 has not been changed since installation.

On Thu, Jun 2, 2022 at 11:42 AM Jessica Meixner @.***> wrote:

UFS error:

[100%] Linking Fortran executable ufs_model /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg24.a(addgrid.f.o): In function `addgrid': /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/addgrid.f:92: undefined reference to for_stop_core_quiet' /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a(compack.f.o): In functioncompack_': /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:382: undefined reference to for_stop_core_quiet' /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:351: undefined reference tofor_stop_core_quiet' /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:156: undefined reference to for_stop_core_quiet' /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:152: undefined reference tofor_stop_core_quiet' /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/g2/3.4.2/lib64/libg2_4.a(compack.f.o):/scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/g2-v3.4.2/src/compack.f:141: more undefined references to `for_stop_core_quiet' follow make[2]: [ufs_model] Error 1 make[1]: [CMakeFiles/ufs_model.dir/all] Error 2 make: *** [all] Error 2

GSI error:

++ /apps/lmod/lmod/libexec/lmod sh load modulefile.ProdGSI.hera Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "w3nco/2.4.1" Try: "module spider w3nco/2.4.1" to see how to load the module(s).

UPP error:

Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "w3nco/2.4.1" Try: "module spider w3nco/2.4.1" to see how to load the module(s).

GLDAS error is unclear

Workflow utils error:

/scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/nemsio/2.5.4/lib64/libnemsio.a(nemsio_openclose.f90.o): In function nemsio_openclose_mp_nemsio_stop_': /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/nemsio-v2.5.4/src/nemsio_openclose.f90:5138: undefined reference tofor_stop_core_quiet' /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/intel-2022.1.2/nemsio/2.5.4/lib64/libnemsio.a(nemsio_openclose.f90.o): In function nemsio_openclose_mp_nemsio_stop_': /scratch2/NCEPDEV/nwprod/hpc-stack/src/develop/pkg/nemsio-v2.5.4/src/nemsio_openclose.f90:5138: undefined reference tofor_stop_core_quiet' make[2]: [fv3nc2nemsio.fd/fv3nc2nemsio.x] Error 1 make[1]: [fv3nc2nemsio.fd/CMakeFiles/fv3nc2nemsio.x.dir/all] Error 2 make[2]: [regrid_nemsio.fd/regrid_nemsio.x] Error 1 make[1]: [regrid_nemsio.fd/CMakeFiles/regrid_nemsio.x.dir/all] Error 2 [ 42%] Built target enkf_chgres_recenter_nc.x make: *** [all] Error 2

@Hang-Lei-NOAA https://github.com/Hang-Lei-NOAA @kgerheiser https://github.com/kgerheiser have there been module updates in hpc-stack on hera recently that might help explain the above errors? @WalterKolczynski-NOAA https://github.com/WalterKolczynski-NOAA any guess on when things last worked in the develop branch of the workflow (time/commit)?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/837#issuecomment-1145011828, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFDCHLSOHGYCIAARGV3VNDI4ZANCNFSM5XVIVPUA . You are receiving this because you were mentioned.Message ID: @.***>