noaa-ocs-modeling / CoastalApp

CoastalApp will be deprecated. Please switch to https://github.com/oceanmodeling/ufs-coastal . CoastalApp is a NUOPC application implemented following UFS best practices to couple coastal ocean models and other domains (Sea Ice, Atmosphere, Wave, Inland Hydrology, ...)
https://github.com/oceanmodeling/ufs-coastal
Creative Commons Zero v1.0 Universal
13 stars 32 forks source link

WW3 no-compile on femto/intel/openmpi, netcdf, workaround #152

Open platipodium opened 1 year ago

platipodium commented 1 year ago

Upon

./build.sh --component "WW3" --plat femto --compiler intel

I get

+ exec make -j 1 WW3_COMP=intel COMP_BINDIR=/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3_INSTALL ww3_nems
make[1]: Entering directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/esmf'

                *****************************
              ***   WAVEWATCH III setup     ***
                *****************************

[INFO] local env file wwatch3.env found in /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/wwatch3.env
   Setup file /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/wwatch3.env found
      Printer (listings)          : unspecified
      auxiliary FORTRAN compiler  : gfortran
      auxiliary C compiler        : gcc
      Source directory            : /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model
      Scratch directory           : /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp
      Save source code            : yes
      Save listings               : yes

   Setup makefile for auxiliary programs

   Compile auxiliary programs
make[2]: Entering directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tools'
gfortran -o /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/w3adc w3adc.f
gfortran -o /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/w3list w3list.f
gfortran -o /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/w3prnt w3prnt.f
gfortran -o /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/w3split w3split.f
make[2]: Leaving directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tools'

   Setup comp & link files
      sed /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/comp.tmpl => /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/comp
      sed /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/link.tmpl => /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/link

   Create required model subdirectories

Finished setting up WAVEWATCH III

\rm -f /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/tempswitch
/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/w3_make ww3_grid

                *****************************
              ***   WAVEWATCH III make      ***
                *****************************

Main directory    : /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model
Scratch directory : /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp
Save source codes : yes
Save listings     : yes
Shared Object     : No

[INFO] list of the programs selected : ww3_grid
Making makefile ...

                *****************************
              ***   WAVEWATCH III makefile  ***
                *****************************

switch = /sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/switch
   Checking all subroutines for modules (this may take a while) ...

Processing ww3_grid
---------------------
make[2]: Entering directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/ftn'
ad3 : processing constants
ad3 : processing scrip_kindsmod [no w3adc]
mv: cannot stat 'scrip_kindsmod.o': No such file or directory
mv: cannot stat 'constants.o': No such file or directory
ad3 : processing scrip_iounitsmod [no w3adc]
ad3 : processing scrip_constants [no w3adc]
ad3 : processing scrip_timers [no w3adc]
ad3 : processing w3servmd
mv: cannot stat 'scrip_constants.o': No such file or directory
mv: cannot stat 'scrip_timers.o': No such file or directory
mv: cannot stat 'scrip_iounitsmod.o': No such file or directory
mv: cannot stat 'w3servmd.o': No such file or directory
ad3 : processing scrip_errormod [no w3adc]
ad3 : processing w3gsrumd
ad3 : processing w3arrymd
mv: cannot stat 'scrip_errormod.o': No such file or directory
mv: cannot stat 'w3gsrumd.o': No such file or directory
ad3 : processing scrip_netcdfmod [no w3adc]
mv: cannot stat 'w3arrymd.o': No such file or directory
ad3 : processing w3gdatmd
mv: cannot stat 'scrip_netcdfmod.o': No such file or directory
ad3 : processing scrip_grids [no w3adc]
mv: cannot stat 'w3gdatmd.o': No such file or directory
ad3 : processing w3odatmd
ad3 : processing w3idatmd
mv: cannot stat 'scrip_grids.o': No such file or directory
ad3 : processing scrip_remap_vars [no w3adc]
mv: cannot stat 'w3odatmd.o': No such file or directory
mv: cannot stat 'w3idatmd.o': No such file or directory
ad3 : processing w3timemd
ad3 : processing w3dispmd
ad3 : processing w3nmlgridmd
mv: cannot stat 'scrip_remap_vars.o': No such file or directory
ad3 : processing wmmdatmd
mv: cannot stat 'w3timemd.o': No such file or directory
mv: cannot stat 'w3dispmd.o': No such file or directory
mv: cannot stat 'w3nmlgridmd.o': No such file or directory
ad3 : processing scrip_remap_conservative [no w3adc]
ad3 : processing w3adatmd
ad3 : processing scrip_remap_write [no w3adc]
mv: cannot stat 'wmmdatmd.o': No such file or directory
ad3 : processing scrip_remap_read [no w3adc]
mv: cannot stat 'scrip_remap_conservative.o': No such file or directory
mv: cannot stat 'scrip_remap_write.o': No such file or directory
mv: cannot stat 'w3adatmd.o': No such file or directory
ad3 : processing w3src4md
ad3 : processing w3triamd
ad3 : processing w3snl1md
mv: cannot stat 'scrip_remap_read.o': No such file or directory
mv: cannot stat 'w3src4md.o': No such file or directory
ad3 : processing w3parall
mv: cannot stat 'w3snl1md.o': No such file or directory
mv: cannot stat 'w3triamd.o': No such file or directory
ad3 : processing scrip_interface
ad3 : processing w3iogrmd
mv: cannot stat 'w3parall.o': No such file or directory
mv: cannot stat 'scrip_interface.o': No such file or directory
ad3 : processing w3wdatmd
mv: cannot stat 'w3iogrmd.o': No such file or directory
ad3 : processing wmscrpmd
mv: cannot stat 'w3wdatmd.o': No such file or directory
mv: cannot stat 'wmscrpmd.o': No such file or directory
ad3 : processing w3gridmd
mv: cannot stat 'w3gridmd.o': No such file or directory
ad3 : processing ww3_grid
mv: cannot stat 'ww3_grid.o': No such file or directory
      Linking ww3_grid
      *** file ww3_grid.o not found ***
      *** file w3wdatmd.o not found ***
      *** file w3gdatmd.o not found ***
      *** file w3adatmd.o not found ***
      *** file w3idatmd.o not found ***
      *** file w3odatmd.o not found ***
      *** file wmmdatmd.o not found ***
      *** file w3parall.o not found ***
      *** file w3triamd.o not found ***
      *** file w3gridmd.o not found ***
      *** file w3src4md.o not found ***
      *** file w3snl1md.o not found ***
      *** file w3iogrmd.o not found ***
      *** file constants.o not found ***
      *** file w3servmd.o not found ***
      *** file w3arrymd.o not found ***
      *** file w3dispmd.o not found ***
      *** file w3gsrumd.o not found ***
      *** file w3timemd.o not found ***
      *** file w3nmlgridmd.o not found ***
      *** file scrip_constants.o not found ***
      *** file scrip_grids.o not found ***
      *** file scrip_iounitsmod.o not found ***
      *** file scrip_remap_vars.o not found ***
      *** file scrip_timers.o not found ***
      *** file scrip_errormod.o not found ***
      *** file scrip_interface.o not found ***
      *** file scrip_kindsmod.o not found ***
      *** file scrip_remap_conservative.o not found ***
      *** file wmscrpmd.o not found ***
      *** file scrip_netcdfmod.o not found ***
      *** file scrip_remap_write.o not found ***
      *** file scrip_remap_read.o not found ***
*** Missing object files ***
make[2]: *** [/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/exe/ww3_grid] Error 3
make[2]: Leaving directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/ftn'
make[1]: *** [gout] Error 1
make[1]: Leaving directory `/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/esmf'
make: *** [/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3_INSTALL/nuopc.mk] Error 2
platipodium commented 1 year ago

I already applied the fix to cmplr.env described in #129

platipodium commented 1 year ago

Is it problematic that it states:

      auxiliary FORTRAN compiler  : gfortran
      auxiliary C compiler        : gcc

Overwriting this in bin/wwatch3.env does not have an effect ...

platipodium commented 1 year ago

Trying:

cd  ~/devel/noaa/CoastalApp-Test/WW3/model/ftn
export aPe=../exe
export aPb=../bin
export aPo=../obj

results in

...
 Exporting WWATCH3_ENV=/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/wwatch3.env
ad3 : processing w3wdatmd
 Exporting WWATCH3_ENV=/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/wwatch3.env
ad3 : processing wmmdatmd
      Linking ww3_grid
 Exporting WWATCH3_ENV=/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/bin/wwatch3.env
make: *** [../exe/ww3_grid] Error 1
platipodium commented 1 year ago

Trying:

make ../exe/ww3_grid

make: *** [../exe/ww3_grid] Error 1

The execution of the code within the target fails:

./bin/link ww3_grid w3wdatmd w3gdatmd w3adatmd w3idatmd w3odatmd wmmdatmd w3parall w3triamd w3gridmd w3src4md w3snl1md w3iogrmd constants w3servmd w3arrymd w3dispmd w3gsrumd w3timemd w3nmlgridmd scrip_constants scrip_grids scrip_iounitsmod scrip_remap_vars scrip_timers scrip_errormod scrip_interface scrip_kindsmod scrip_remap_conservative wmscrpmd scrip_netcdfmod scrip_remap_write scrip_remap_read || echo fail
platipodium commented 1 year ago

.. continuting ..

This creates the call

ifort -o ww3_grid -g -O1 -xhost ww3_grid.o w3wdatmd.o w3gdatmd.o w3adatmd.o w3idatmd.o w3odatmd.o wmmdatmd.o w3parall.o w3triamd.o w3gridmd.o w3src4md.o w3snl1md.o w3iogrmd.o constants.o w3servmd.o w3arrymd.o w3dispmd.o w3gsrumd.o w3timemd.o w3nmlgridmd.o scrip_constants.o scrip_grids.o scrip_iounitsmod.o scrip_remap_vars.o scrip_timers.o scrip_errormod.o scrip_interface.o scrip_kindsmod.o scrip_remap_conservative.o wmscrpmd.o scrip_netcdfmod.o scrip_remap_write.o scrip_remap_read.o

in ../obj

platipodium commented 1 year ago

Manually changing to ../obj and executing

ifort -o ww3_grid -g -O1 -xhost ww3_grid.o w3wdatmd.o w3gdatmd.o w3adatmd.o w3idatmd.o w3odatmd.o wmmdatmd.o w3parall.o w3triamd.o w3gridmd.o w3src4md.o w3snl1md.o w3iogrmd.o constants.o w3servmd.o w3arrymd.o w3dispmd.o w3gsrumd.o w3timemd.o w3nmlgridmd.o scrip_constants.o scrip_grids.o scrip_iounitsmod.o scrip_remap_vars.o scrip_timers.o scrip_errormod.o scrip_interface.o scrip_kindsmod.o scrip_remap_conservative.o wmscrpmd.o scrip_netcdfmod.o scrip_remap_write.o scrip_remap_read.o

reveals that netcdf is the culprit (again)

/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp/w3gridmd.F90:3368: undefined reference to `netcdf_mp_nf90_create_'
/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp/w3gridmd.F90:3369: undefined reference to `netcdf_mp_nf90_def_dim_'
/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp/w3gridmd.F90:3370: undefined reference to `netcdf_mp_nf90_def_dim_'
/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp/w3gridmd.F90:3371: undefined reference to `netcdf_mp_nf90_def_dim_'
/sciclone/home20/clemmen/devel/noaa/CoastalApp-Test/WW3/model/tmp/w3gridmd.F90:3373: undefined reference to `netcdf_mp_nf90_def_var_manydims_'
...
platipodium commented 1 year ago

This needs the fix mentioned in #102

sed 's/`$NETCDF_CONFIG --fc`/gfortran/g' -i WW3/model/bin/comp.tmpl
sed 's/`$NETCDF_CONFIG --includedir`/\/usr\/local\/include/g' -i WW3/model/bin/comp.tmpl
sed 's/`$NETCDF_CONFIG --fc`/gfortran/g' -i WW3/model/bin/link.tmpl
sed 's/`$NETCDF_CONFIG --includedir`/\/usr\/local\/include/g' -i WW3/model/bin/link.tmpl
sed 's/`$NETCDF_CONFIG --flibs`/-L\/usr\/local\/lib -lnetcdff -lnetcdf /g' -i WW3/model/bin/link.tmpl

Specifically with recently updated code, it is only one line in each file

diff --git a/model/bin/comp.tmpl b/model/bin/comp.tmpl
index feba730..6784ba9 100755
--- a/model/bin/comp.tmpl
+++ b/model/bin/comp.tmpl
@@ -79,8 +79,9 @@
   # netcdf include dir
   if [ "$netcdf_compile" = 'yes' ]
   then
-    if [ "$mpi_mod" = 'no' ]; then comp="`$NETCDF_CONFIG --fc`"; fi
-    opt="$opt `$NETCDF_CONFIG --cflags`"
+    if [ "$mpi_mod" = 'no' ]; then comp="ifort"; fi
+    #opt="$opt `$NETCDF_CONFIG --cflags`"
+    opt="$opt -I/usr/local/skylake/linux-centos7-x86_64/intel-18.0.5/netcdf-fortran-4.4.4-rbpzwccr65wvutgic74rwjg6azlynglt/include -I/usr/local/skyla
   fi

   # ftn include dir
diff --git a/model/bin/link.tmpl b/model/bin/link.tmpl
index 24c081f..5a3262e 100755
--- a/model/bin/link.tmpl
+++ b/model/bin/link.tmpl
@@ -113,8 +113,10 @@

   # netcdf library dir
   if [ "$netcdf_compile" = 'yes' ] ; then
-    if [ "$mpi_mod" = 'no' ]; then comp="`$NETCDF_CONFIG --fc`"; fi
-    libs="$libs `$NETCDF_CONFIG --flibs` `$NETCDF_CONFIG --libs`"
+    if [ "$mpi_mod" = 'no' ]; then comp="ifort"; fi
+    #libs="$libs `$NETCDF_CONFIG --flibs` `$NETCDF_CONFIG --libs`"
+    libs="$libs -L/usr/local/skylake/linux-centos7-x86_64/intel-18.0.5/netcdf-fortran-4.4.4-rbpzwccr65wvutgic74rwjg6azlynglt/lib -lnetcdff -L/usr/loc
+
   fi
saeed-moghimi-noaa commented 1 year ago

@aliabdolali @pvelissariou1

Would you please chime in!?

Thanks, -Saeed

platipodium commented 1 year ago

My idea is that much of this and other problems are related to non-portable passing of environment variables ... like the $NETCDF_CONFIG here.

aliabdolali commented 1 year ago

@saeed-moghimi-noaa @platipodium we have implemented cmake in the latest version of WW3, so no need to setting such environmental variables anymore. I would recommend keeping this non-portable passing of env for a while until we merge the top of WW3 to the COASTAL App, then life is much easier.

platipodium commented 1 year ago

@aliabdolali CMake is good news! Do you have a time line on merging WW3 HEAD to Coastal App?

aliabdolali commented 1 year ago

@aliabdolali CMake is good news! Do you have a time line on merging WW3 HEAD to Coastal App? My understanding from our conversation with NOS colleagues is that we want to have everything up and running with different configurations (including regtests) and then we will plan for further development.