noaa-ocs-modeling / CoastalApp

CoastalApp will be deprecated. Please switch to https://github.com/oceanmodeling/ufs-coastal . CoastalApp is a NUOPC application implemented following UFS best practices to couple coastal ocean models and other domains (Sea Ice, Atmosphere, Wave, Inland Hydrology, ...)
https://github.com/oceanmodeling/ufs-coastal
Creative Commons Zero v1.0 Universal
13 stars 31 forks source link

CoastalApp compilation on Cheyenne fails #156

Open pvelissariou1 opened 1 year ago

pvelissariou1 commented 1 year ago
Ufuk Turuncoglu Ufuk Turuncoglu Jan 27, 2023, 1:16 PM (3 days ago)    

Ufuk Turuncoglu to Saeed, me, Daniel | to Saeed, me, Daniel to Saeed, me, Daniel

Ufuk Turuncoglu Jan 27, 2023, 1:16 PM (3 days ago) to Saeed, me, Daniel

Hi All,

I am following steps in

https://github.com/noaa-ocs-modeling/CoastalApp-testsuite

to run the tests. I am getting following error when I try to compile the model on Cheyenne. It seems that we have module files under modulefiles/ directory.

./build.sh --compiler intel --platform cheyenne --component "atmesh pahm adcirc ww3" -y envmodules_intel.cheyenne :: This environment is either not configured or not supported Exiting ...

Anyway, I just want to let you know. If you have no idea why it is not working I could try to fix it but I just want o ask you first. I also tried with gnu and it is not working too. Any idea? I could also try on Orion if you want.

Regards,

—ufuk

Ufuk Turuncoglu Jan 27, 2023, 1:17 PM (3 days ago) to Saeed, me, Daniel

Do i also need to follow the steps in compile-NEMS.x.txt (under CoastalApp directory)?

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov Jan 27, 2023, 1:29 PM (3 days ago) to Ufuk, Saeed, Daniel

Ufuk hi,

I haven't configured CoastalApp for "cheyenne" because I don't have access to this platform. What you can do is go into CoastalApp/modulefiles, copy the hera modulefiles to cheyenne and modify them according to cheyenne's module configuration. I think it will be better if we could have a chat (google meet) to discuss details. Are you free today?

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu Jan 27, 2023, 1:36 PM (3 days ago) to me, Saeed, Daniel

Hi,

I have just comment you the section in the module file (for Cheyenne, Intel) that causes issue. Now I am trying to build and see how it goes. If that won’t work I’ll try your suggestion and use Hera one as a base. I’ll let you know if I need help about it and then we could have a chat. Is there any other platform that is tested regularly such as Orion?

Thanks for your help,

—ufuk

Ufuk Turuncoglu Jan 27, 2023, 1:38 PM (3 days ago) to me, Saeed, Daniel

BTW, which Here module is the best one for Intel compiler. I could see multiple under module files/ directory. Thanks.

—ufuk

On Jan 27, 2023, at 12:29 PM, Panagiotis Velissariou - NOAA Affiliate [panagiotis.velissariou@noaa.gov](mailto:panagiotis.velissariou@noaa.gov) wrote:

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov Jan 27, 2023, 1:49 PM (3 days ago) to Ufuk

envmodules_intel.hera is the file currently used. Basically the code in all modulefiles is the same, only the modules that are loaded are different between platforms. Could you please load the module manually and send me what NETCDF and HDF environment variables are set?

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu Jan 27, 2023, 3:11 PM (3 days ago) to me

Hi,

I did following,

cc = mpicc cxx = mpicxx

To

if [[ -z "${CC}" ]]; then cc = mpicc else cc = ${CC} fi if [[ -z "${CXX}" ]]; then cxx = mpicxx else cxx = ${CXX} fi

Since the variables can be provided by the environment variables.

module purge module load cmake/3.22.0 module load ncarenv/1.3 module load intel/2022.1 module load mpt/2.25 module load ncarcompilers/0.5.0 module use /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1 module load hpc-mpt/2.25 export CC=mpicc export PCC=mpicc export CXX=mpicxx export PCXX=mpicxx export FC=mpif90 module load zlib/1.2.11 module load hdf5/1.10.6 module load netcdf/4.7.4 module load pio/2.5.7 module load esmf/8.3.0b09

When I run the script using following command,

./build.sh --compiler intel --platform cheyenne --component "atmesh pahm adcirc ww3" --tp parmetis -y

I am getting following error,

make[1]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/metis/build/Linux-x86_64' Makefile:11: *** missing separator. Stop. ERROR:: compileMetis (main): called from: functions_build the configuration of the ParMETIS library source failed tried to run the make command as: make config CC=mpicc CXX=mpicxx cc=mpicc cxx=mpicxx CFLAGS= CONFIG_FLAGS1=-DCMAKE_C_FLAGS='"-DIDXTYPEWIDTH=32 -DREALTYPEWIDTH=32"' gklib_path=/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/metis/GKlib metis_path=/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/metis prefix=/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/del Exiting now …

I am still investigating. I’ll update you when I could compile the model then maybe you might want to push those changes to the main repository or I could create a branch and we could have a PR later.

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov Jan 27, 2023, 3:38 PM (3 days ago) to Ufuk

You don't need to modify any of these in the Makefile. All these are set by the build script. Check the file scripts/functions_build where there is a function compileMetis that defines all these variables and configures metis/parmetis properly. It should work in your case, if not please let me know the errors you are getting.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu Jan 29, 2023, 11:47 PM (12 hours ago) to me

Hi,

I past that issue but I am having another error as following,

-- Configuration on "cheyenne" waits verification. -- Found NetCDF: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include (found version "4.7.4") found components: C Fortran -- FindNetCDF defines targets: -- - NetCDF_VERSION [4.7.4] -- - NetCDF_PARALLEL [TRUE] -- - NetCDF_C_CONFIG_EXECUTABLE [/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/bin/nc-config] -- - NetCDF::NetCDF_C [STATIC] [Root: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4] Lib: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdf.a -- - NetCDF_Fortran_CONFIG_EXECUTABLE [/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/bin/nf-config] -- - NetCDF::NetCDF_Fortran [STATIC] [Root: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4] Lib: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdff.a CMake Error at cmake/check_netcdf.cmake:143 (message): The NetCDF library specified is not compatible with the specified compilers.It will not be enabled. Specify a different path or disable NetCDF.Ensure that you specify the same compilers to build PAHM as were used to build the NetCDF library.

Call Stack (most recent call first): CMakeLists.txt:238 (include)

-- Configuring incomplete, errors occurred!

and then

pahm_mod.F90(15): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [PAHM_MESH] USE PaHM_Mesh ------^ pahm_mod.F90(107): error #6404: This name does not have a type, and must have an explicit type. [NP] theData%numNd = np --------------------^ pahm_mod.F90(108): error #6404: This name does not have a type, and must have an explicit type. [NE] theData%numEl = ne --------------------^ pahm_mod.F90(122): error #6404: This name does not have a type, and must have an explicit type. [SLAM] theData%ndCoords((iCnt - 1) nDims + 1) = slam(iCnt) -------------------------------------------------^ pahm_mod.F90(123): error #6404: This name does not have a type, and must have an explicit type. [SFEA] theData%ndCoords((iCnt - 1) nDims + 2) = sfea(iCnt) -------------------------------------------------^ pahm_mod.F90(130): error #6404: This name does not have a type, and must have an explicit type. [NM] theData%elConnect((iCnt - 1) * numFaceEl + 1) = nm(iCnt, 1) ------------------------------------------------------^ pahm_mod.F90(136): error #6404: This name does not have a type, and must have an explicit type. [DP] theData%bathymetry = dp -------------------------^ compilation aborted for pahm_mod.F90 (code 1)

It seems that those parameters (i.e., np) is defined in PAHM/src/mesh.F90 and it needs to be compiled before pahm_mod.F90. Maybe it is because if netcdf issue, not sure.

At this point I put following to modulefiles/envmodules_intel.cheyenne (trying to use exact modules used for UFS),

module load cmake/3.22.0 module load ncarenv/1.3 module load intel/2022.1 module load mpt/2.25 module load ncarcompilers/0.5.0 module use /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1 module load hpc-mpt/2.25 export CC=mpicc export PCC=mpicc export CXX=mpicxx export PCXX=mpicxx export FC=mpif90 export F90=mpif90 export PFC=mpif90 export PF90=mpif90 module load zlib/1.2.11 module load hdf5/1.10.6 module load netcdf/4.7.4 module load pio/2.5.7 module load esmf/8.3.0b09

So, maybe there are some restriction in CostallApp side in terms of module versions. In my case, I am using exact modules and netcdf needs to be consistent with compiler. Maybe there is something in PHMS side. I’ll try to find the source. In the meantime, if you have any suggestion just let me know.

Thanks,

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov 9:00 AM (3 hours ago) to Ufuk

Hi Ufuk,

I have fixed these issues, there was a conflict with the schism model. Let me update the github repo and I'll let you know.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov 10:15 AM (2 hours ago) to Ufuk

Hi,

Please clone the develop-updates branch for the time being: git clone --recurse-submodules https://github.com/noaa-ocs-modeling/CoastalApp.git -b develop-updates

With this branch and your module file, it should compile on cheyenne. Please let me know.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu 10:43 AM (1 hour ago) Thanks for the quick fix. I’ll try and let you know.

Ufuk Turuncoglu 10:58 AM (1 hour ago) to me

Hi Again,

I am still getting same error with your branch,

-- - NetCDF::NetCDF_Fortran [STATIC] [Root: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4] Lib: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdff.a CMake Error at cmake/check_netcdf.cmake:143 (message): The NetCDF library specified is not compatible with the specified compilers.It will not be enabled. Specify a different path or disable NetCDF.Ensure that you specify the same compilers to build PAHM as were used to build the NetCDF library.

Call Stack (most recent call first): CMakeLists.txt:238 (include)

-- Configuring incomplete, errors occurred! See also "/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeOutput.log". Couldn't run "make" due to "cmake" errors. make[1]: Entering directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/nuopc'

mpif90 -c -g -traceback -fp-model precise -fPIC -O2 -fPIC -assume realloc_lhs -m64 -mcmodel=small -pthread -threads -qopenmp -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/esmf/8.3.0b09/mod -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/esmf/8.3.0b09/include -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_VERSION_STRING_GIT='v8.3.0b09' -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_MPIIO -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DSx86_64_small=1 -DESMF_OS_Linux=1 -DESMF_COMM=mpt -DESMF_DIR=/glade/work/jongkim/stacks/hash/hpc-stack-esmf/pkg/v8.3.0b09 -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -I. -I/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL -I/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/include -L/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib -L. -L/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL -L/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/lib -DWITHPETLISTS_on -DESMF_VERSION_MAJOR=8 pahm_mod.F90 pahm_mod.F90(15): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [PAHM_MESH] USE PaHM_Mesh ------^ pahm_mod.F90(107): error #6404: This name does not have a type, and must have an explicit type. [NP] theData%numNd = np --------------------^ pahm_mod.F90(108): error #6404: This name does not have a type, and must have an explicit type. [NE] theData%numEl = ne --------------------^ pahm_mod.F90(122): error #6404: This name does not have a type, and must have an explicit type. [SLAM] theData%ndCoords((iCnt - 1) nDims + 1) = slam(iCnt) -------------------------------------------------^ pahm_mod.F90(123): error #6404: This name does not have a type, and must have an explicit type. [SFEA] theData%ndCoords((iCnt - 1) nDims + 2) = sfea(iCnt) -------------------------------------------------^ pahm_mod.F90(130): error #6404: This name does not have a type, and must have an explicit type. [NM] theData%elConnect((iCnt - 1) * numFaceEl + 1) = nm(iCnt, 1) ------------------------------------------------------^ pahm_mod.F90(136): error #6404: This name does not have a type, and must have an explicit type. [DP] theData%bathymetry = dp -------------------------^ compilation aborted for pahm_mod.F90 (code 1) Makefile:48: recipe for target 'pahm_mod.o' failed make[1]: [pahm_mod.o] Error 1 make[1]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/nuopc' /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/NEMS/src/incmake/component_PAHM.mk:30: recipe for target '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/pahm.mk' failed make: [/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/pahm.mk] Error 2

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov 11:56 AM (26 minutes ago) to Ufuk

Hi,

Unfortunately CoastalApp does not work with the hpc-stack static libraries. Can you use the OS supplied libraries for NetCDF/HDF5 and ESMF so all dependencies are resolved? Let me check on hera to see what compiles in CoastalApp using hpc-stack. For sure ADCIRC can not be compiled using hpc-stack. It is in the to-do list to add the hpc-stack functionality in CoastalApp.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu 12:00 PM (21 minutes ago) to me

Thanks for your help. Okay. I could try with others frowned by the system. In any case we need to fix this issue to make components work under UFS since they are using theses modules. If you could make it work under Hera just let me know. One more question, I think set of libraries are used by CostallApp that could be part of hpc-stack. Could you list them for me? I know parmetis is one of them, which I think you don’t need to compile every time with the app itself and could be part of the hpc-stack (or spack-stack in the near future). Right?

—ufuk

Panagiotis Velissariou - NOAA Affiliate 12:15 PM (7 minutes ago) to Ufuk

ParMetis can not be part of hpc-stack due to licensing issues. We had this discussion with NCEP and it was decided that ParMetis can not be part of the official NOAA modeling infrastructure due to its license. This is the reason I have the ParMetis functionality in CoastalApp. Now, WW3 is moving away from ParMetis (I don't know when this will be done) and SCHISM has the capability to use an externally compiled ParMetis or NOT.

As long as you compiled ParMetis once in CoastalApp (it is installed in CoastalApp/THIRDPARTY_INSTALL) you can rerun the build.sh scripts as: PARMETISHOME=FULL_PATH_TO_CoastalApp/THIRDPARTY_INSTALL ./build.sh "your options" (without the --tp parmetis option) and CoastalApp will pick-up the installed ParMetis. Just make sure that all libraries, including ParMetis, are compiled using the same compilers and mpi libraries.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu 12:14 PM (7 minutes ago) to me

I also tried to do this to see why netcdf is failing. I write the INFO variables from Cmake build (add message(INFO ${NetCDF3_LOG}) to PAHM/cmake/check_netcdf.cmake) and it seems that it is failing like following,

Change Dir: /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_95f2d/fast && gmake[1]: Entering directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp' /usr/bin/gmake -f CMakeFiles/cmTC_95f2d.dir/build.make CMakeFiles/cmTC_95f2d.dir/build gmake[2]: Entering directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp' Building Fortran object CMakeFiles/cmTC_95f2d.dir/netcdf3check.f90.o /glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/2022.1/mpi/mpif90 -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -extend-source 132 -heap-arrays 0 -fno-alias -sox -qno-opt-dynamic-align -O2 -debug minimal -nowarn -fp-model source -assume byterecl -mcmodel=medium -c /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/netcdf3check.f90 -o CMakeFiles/cmTC_95f2d.dir/netcdf3check.f90.o gmake[2]: No rule to make target '/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/hdf5/1.10.6/lib -lhdf5_hl -lhdf5 /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/zlib/1.2.11/lib -lz -ldl -lm -lnetcdf -lm', needed by 'cmTC_95f2d'. Stop. gmake[2]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp' Makefile:127: recipe for target 'cmTC_95f2d/fast' failed gmake[1]: [cmTC_95f2d/fast] Error 2 gmake[1]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp'

Then, I copy the code that test netcdf3 and run the command manually.

/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/2022.1/mpi/mpif90 -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -extend-source 132 -heap-arrays 0 -fno-alias -sox -qno-opt-dynamic-align -O2 -debug minimal -nowarn -fp-model source -assume byterecl -mcmodel=medium -c netcdf3check.f90

and it compiled code without any issue. It is also same netcdf4 one.So, I am not sure why this is not working under CMake build. I think it is not related with the hpc-stack modules. Anyway, I’ll try the find the source of the issue.

—ufuk

Ufuk Turuncoglu 12:20 PM (2 minutes ago) to me

Hi,

Okay. It is good to know the background about the parmetis. This will help when we start to move the components to UFS. Thanks.

—ufuk

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> Yes, this is the idea to have CDEPS handle everything. Can CDEPS read the input data (data + mesh) and do the process downloading->interpolating on the fly? What pre-processing steps are required from us (if any) before involving CDEPS?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Feb 21, 2023 at 11:43 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. That is great. If I could create mesh file for the original data file then CDEPS will handle rest.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1438870407, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7HVFVXEJM7YN5TEPTWYT5CFANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Yes, CDEPS could read ESMf mesh file and data file without any issue (it uses PIO in the backend). The interpolation can be also done in CDEPS is data component and stream meshes are different. You could also perform interpolation outside of CDEPS (we will use this for ATMMESH case) using NUOPC connectors or CMEPS. If you could point me the data file, then I could check it to see what is required. If it is on simple lat-lon grid or curvilinear then I could create ESMF Mesh file by using NCL SCRIP capability and ESMF offline mesh generator tool. Downloading is not part of CDPES. That needs to be implemented in the workflow level (ufs-coastal-app) if you want to create easy to use application. I think workflow design and implementation is not part of this project but once we have all the component under UFS and if we still time for it, we could look at it.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.**> The most common and widely used atmospheric model grids are latitude-longitude square/rectangular/curvilinear* grids. The latitude/longitude dimensions come in two flavors as 1-D vectors and as 2-D arrays.

My suggestion is to target these types of grids for atmospheric models/data and to focus on a lat/lon or latitude/longitude naming scheme for the spatial dimensions of the grid.

Most of the atmospheric model NetCDF outputs comply with the CF standards (see https://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.pdf), but this is not always the case.

Furthermore, we need to make sure that the longitude data are in -180.0 degrees to +180.0 degrees notation. If the longitudes are in the 0-360 degrees notation they will need to be converted (I do have the conversion code).

Here are some data files for the atmospheric fields stored on orion: 1) Florence_HRRR.nc (High-Resolution Rapid Refresh - HRRR) that uses the 2-D latitude/longitude arrays Location: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/Florence_HRRR.nc 2) Florence_HWRF.nc (Hurricane Weather Research & Forecasting Model - HWRF) that uses the 1-D latitude/longitude vectors Location: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/Florence_HWRF.nc 3) Florence_CFS.nc (Climate Forecast System - CFS) that uses the 1-D latitude/longitude vectors Location: /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/Florence_CFS.nc

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Feb 21, 2023 at 11:58 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Yes, CDEPS could read ESMf mesh file and data file without any issue (it uses PIO in the backend). The interpolation can be also done in CDEPS is data component and stream meshes are different. You could also perform interpolation outside of CDEPS (we will use this for ATMMESH case) using NUOPC connectors or CMEPS. If you could point me the data file, then I could check it to see what is required. If it is on simple lat-lon grid or curvilinear then I could create ESMF Mesh file by using NCL SCRIP capability and ESMF offline mesh generator tool. Downloading is not part of CDPES. That needs to be implemented in the workflow level (ufs-coastal-app) if you want to create easy to use application. I think workflow design and implementation is not part of this project but once we have all the component under UFS and if we still time for it, we could look at it.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1438889143, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP3I56UBVUU4BFKS7PLWYT64PANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Thanks. Under /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/ I could only see the CFS data not others. BTW, which one do I need to use for florence_hsofs.atm2adc test case.

pvelissariou1 commented 1 year ago

All of them are for Florence but not on the hsofs mesh. The data are in a rectangular/square grid that will need to be interpolated into the hsofs mesh via CDEPS. Only Florence_CFS.nc is on orion now. I am uploading Florence_HWRF.nc now (it is going to take about 12 hrs). Florence_HWRF.nc was used to create the Florence_HWRF_HRRR_HSOFS.nc data, so you may use that one.Florence_CFS.nc

I thought for our purposes anyone of these files will do. Why don't you start with Florence_CFS.nc and interpolate to the HSOFS mesh?

Your suggestion?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Feb 21, 2023 at 11:48 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Thanks. Under /work/noaa/nosofs/pvelissa/for_Ufuk_atm_data/ I could only see the CFS data not others. BTW, which one do I need to use for florence_hsofs.atm2adc test case.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1439473852, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4VAWRSZMCFI2WFM2DWYWSDZANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. That sounds fine. I could try with Florence_CFS.nc. Thanks.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> I have started uploading the files to Orion using globus. It should take a few hours.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Feb 22, 2023 at 12:07 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. That sounds fine. I could try with Florence_CFS.nc. Thanks.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1439486768, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6PVTELFQRFE2BPSU3WYWUJLANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> I uploaded the data file Florence_HWRF.nc successfully on orion. Florence_HRRR.nc failed due to network issues. You might want to use Florence_HWRF.nc for CDEPS.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Feb 22, 2023 at 1:32 AM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

@Ufuk Turuncoglu @.***> I have started uploading the files to Orion using globus. It should take a few hours.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Feb 22, 2023 at 12:07 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. That sounds fine. I could try with Florence_CFS.nc. Thanks.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1439486768, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP6PVTELFQRFE2BPSU3WYWUJLANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 As you already know that the dataset (Florence_CFS.nc) was not covering the simulation dates and CDEPS was unable to provide coupling files to ADCIRC. If you don't mind could you upload data for florence_hsofs.atm2adc case to Orion. It started from,

start_year:              2008
start_month:             9
start_day:               7
start_hour:              0
start_minute:            0
start_second:            0

Then, I could test it again. Thanks.

pvelissariou1 commented 1 year ago

Hi Ufuk,

Sorry, my mistake. I will generate the data shortly.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Mar 5, 2023 at 10:53 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 As you already know that the dataset (Florence_CFS.nc) was not covering the simulation dates and CDEPS was unable to provide coupling files to ADCIRC. If you don't mind could you upload data for florence_hsofs.atm2adc case to Orion. It started from,

start_year: 2008 start_month: 9 start_day: 7 start_hour: 0 start_minute: 0 start_second: 0

Then, I could test it again. Thanks.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1455446501, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPY5F2YV3WDB6BG2D5DW2VUTNANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

Ufuk,

In florence_hsofs.atm2adc I have for model_configure the following: start_year: 2018 start_month: 09 start_day: 07 start_hour: 0 start_minute: 0 start_second: 0 nhours_fcst: 168

so the data should work. Could you clone the latest CoastalApp-testsuite: git clone https://github.com/noaa-ocs-modeling/CoastalApp-testsuite.git -b develop to make sure you have the correct files?

Thanks Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Mar 6, 2023 at 8:48 AM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

Hi Ufuk,

Sorry, my mistake. I will generate the data shortly.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Mar 5, 2023 at 10:53 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 As you already know that the dataset (Florence_CFS.nc) was not covering the simulation dates and CDEPS was unable to provide coupling files to ADCIRC. If you don't mind could you upload data for florence_hsofs.atm2adc case to Orion. It started from,

start_year: 2008 start_month: 9 start_day: 7 start_hour: 0 start_minute: 0 start_second: 0

Then, I could test it again. Thanks.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1455446501, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPY5F2YV3WDB6BG2D5DW2VUTNANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 okay. maybe I used module_configure from other test florence_shinnecock.atm2adc and forgot to change the dates. Let me try again. Thanks.

uturuncoglu commented 1 year ago

@pvelissariou1 It seems it is working now. My run directory is in /glade/scratch/turuncu/COASTAL_APP/florence_hsofs.atm2adc on Cheyenne. How can I check the results? I could not see the output.

uturuncoglu commented 1 year ago

Maybe it is in PE folders and fort.16 file for each PET but I think I need to have a tool to convert them to netcdf etc.

pvelissariou1 commented 1 year ago

In the run directory don't you see files fort.63.nc, fort.64.nc, etc? If not fort.15 needs to be modified to write out these files. Let me know.

On Monday, March 6, 2023, Ufuk Turunçoğlu @.***> wrote:

Maybe it is in PE folders and fort.16 file for each PET but I think I need to have a tool to convert them to netcdf etc.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci6.googleusercontent.com/proxy/4wySg-9r_DPehdKN6onvfi2ZHzkB6VmcVYdiIUxiCcxr6CyoaxovytswnenVVeRRw5o7R-6moC_VFizetsMnJE0evbzN47eT2DisDybf4aKrdrlWqced3DsBRdhIlOkqKil_ZcskZwQ2BlOneP2qpFcrPqhadBzwVRmWEnd0kdpQ33eNTOXWvVqn_v81FudlnmA8unCJbCWMHGa_xX6S4a67Sy6ckpCP-VyNWIYh_d2MZoWHAA=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TP34W532YR5B3BU5L33W2ZJOXA5CNFSM6AAAAAAULPOB22WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSW3CCYE.gif>Message ID: @.***>

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

uturuncoglu commented 1 year ago

@pvelissariou1 I could see fort.63.nc and fort.64.nc.

uturuncoglu commented 1 year ago

I check those files their time dimensions shows 0 and they are also 1d files (I think on ADCIRC mesh) and it is not possible for me to check them visually.

uturuncoglu commented 1 year ago

If you also check the out file it seems that run ends without any issue. BTW, I change the configuration to run 24 hours. Do I need to change anything in the ADCIRC configuration?

uturuncoglu commented 1 year ago

BTW, the ADCIRC configuration might need to have some tweaks. The CFS data is 6-hourly but WTIMINC is indicated as hourly infort.15. Also it says that it is 7 days of run. So, maybe I need to set RNDAY to 1. Anyway, we might need to modify this file and maybe others to fit to this configuration. Then, I could start to define it as a UFS RT.

pvelissariou1 commented 1 year ago

Yes, I will check fort.15 and the modifications. Give me some time my computer is down.

On Monday, March 6, 2023, Ufuk Turunçoğlu @.***> wrote:

If you also check the out file it seems that run ends without any issue. BTW, I change the configuration to run 24 hours. Do I need to change anything in the ADCIRC configuration?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci4.googleusercontent.com/proxy/1ti6T8zutisUyTJ856hllYD9mMFX6aSUilFpqMhqKK910ucpRd4X2lgM7HBs1-5EF6_gm1Ha8Eq6bu-IShwNyvwq11IJjaFUVBLM_ZK8FkQsgQdGcn6yF3x82Xr-gAH2dwt_k1JoClpiLi0B0iGo0ZnCAHW5eAefDEkYFe9Cs56SFfEOBYfZuH5SVqluVv53oPhavr5mGiL_uoS20VlTwbnbTHqMIuJw2hwBBvm03yGk3p_73Q=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TPZK57Y4XHGQO277J2LW2ZKWLA5CNFSM6AAAAAAULPOB22WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSW3C25S.gif>Message ID: @.***>

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

uturuncoglu commented 1 year ago

@pvelissariou1 Sure. BTW, if you want to use higher temporal resolution dataset to force ADCIRC, we could replace the CFS data after creating RT and validate that ADCIRC producing reasonable results.

pvelissariou1 commented 1 year ago

Ufuk,

For the CFS data change the wind temporal resolution to 6 hrs x 3600s/hr = 21600 s Line: FROM: 3600 3600 ! WTIMINC - meteorological data time increment, RSTIMINC wave forcing increment TO: 21600 3600 ! WTIMINC - meteorological data time increment, RSTIMINC wave forcing increment

I also changed: FROM: -5 0.0 7.0 1800 ! NOUTGE,TOUTSGE,TOUTFGE,NSPOOLGE : GLOBAL ELEVATION OUTPUT INFO (UNIT 63) -5 0.0 7.0 1800 ! NOUTGV,TOUTSGV,TOUTFGV,NSPOOLGV : GLOBAL VELOCITY OUTPUT INFO (UNIT 64) -5 0.0 7.0 1800 ! NOUTGW,TOUTSGW,TOUTFGW,NSPOOLGW : GLOBAL WIND OUTPUT INFO TO: -5 0.0 1000.0 1800 ! NOUTGE,TOUTSGE,TOUTFGE,NSPOOLGE : GLOBAL ELEVATION OUTPUT INFO (UNIT 63) -5 0.0 1000.0 1800 ! NOUTGV,TOUTSGV,TOUTFGV,NSPOOLGV : GLOBAL VELOCITY OUTPUT INFO (UNIT 64) -5 0.0 1000.0 1800 ! NOUTGW,TOUTSGW,TOUTFGW,NSPOOLGW : GLOBAL WIND OUTPUT INFO

You should be ok now.

For RNDAY just leave it as is, it doesn't hurt.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Mar 6, 2023 at 4:00 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Sure. BTW, if you want to use higher temporal resolution dataset to force ADCIRC, we could replace the CFS data after creating RT and validate that ADCIRC producing reasonable results.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1457072391, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZHRE4XZXN4RMEKV7LW2ZM6RANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. I modified the parameter file and run again. I still don't know how I could check the results. Could you give me some information.

pvelissariou1 commented 1 year ago

Let me see what people here use. I'll come back to you. My personal plotting packages are not suitable.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Mar 6, 2023 at 11:39 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. I modified the parameter file and run again. I still don't know how I could check the results. Could you give me some information.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1457571009, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7GK2NYUPN2QSEHCT3W23CX5ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

Hi Ufuk,

I have used FigureGen in the past ( https://ccht.ccee.ncsu.edu/figuregen-v-49/) which is easy to use. Saeed has a python package to plot ADCIRC results ( https://github.com/saeed-moghimi-noaa/ca_adcirc_plot) and compare the outputs with observations. I asked the ADCIRC folks if they have code to quickly plot ADCIRC results (and if they can share), waiting ...

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Mar 7, 2023 at 11:11 AM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

Let me see what people here use. I'll come back to you. My personal plotting packages are not suitable.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Mar 6, 2023 at 11:39 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. I modified the parameter file and run again. I still don't know how I could check the results. Could you give me some information.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1457571009, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7GK2NYUPN2QSEHCT3W23CX5ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 okay. let me check those tools. If the output looks fine then I'll define this case ad RT. Do you have any preference for the next configuration?

pvelissariou1 commented 1 year ago

You mean component? I would suggest SCHISM and then WW3.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Mar 8, 2023 at 11:40 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 okay. let me check those tools. If the output looks fine then I'll define this case ad RT. Do you have any preference for the next configuration?

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1460564549, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4AL765LYOTNQVWDZLW3C763ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

Okay. I'll look at first SCHISM after this.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> Attached is a matlab script to plot water levels from the ADCIRC outputs.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Mar 8, 2023 at 11:45 AM Ufuk Turunçoğlu @.***> wrote:

Okay. I'll look at first SCHISM after this.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1460575726, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7FWBJNPNCCH5HOWXTW3DAU7ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I can't see the attachment.

uturuncoglu commented 1 year ago

@pvelissariou1 i am having trouble to use Python package. I download the simple datasets and try to point my run directory but run plot_maps.py does not produce anything just following output,

rm: cannot remove 'base_info.pyc': No such file or directory
rm: cannot remove 'geo_regions.pyc': No such file or directory
/glade/scratch/turuncu/COASTAL_APP/florence_hsofs.atm2adc/
/glade/scratch/turuncu/COASTAL_APP/ca_adcirc_plot/plot_maps.py:461: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
  dates1 = netCDF4.num2date(ncv1['time'][:],ncv1['time'].units)
[info]: Fin

I need to create very simple plots that shows the run is fine.

uturuncoglu commented 1 year ago

@pvelissariou1 @saeed-moghimi-noaa I added coastal_florence_hsofs_atm2adc as a regression test under UFS. I did not check the results yet since I have issue with plotting packages and I need to have very simple tool (Python scripts etc) to create some plots to verify it is working as expected. Maybe I could try to create it with using NCL etc.

Anyway, the regression test basically runs with NUOPC connectors not CMEPS until the issue in ADCIRC cap is solved. I also created new regression file for coastal app (tests/rt_coastal.conf). We could move the tests that we want to test regularly with UFS to the rt.conf later. All the work is in the following UFS Weather Model fork.

https://github.com/uturuncoglu/ufs-weather-model/tree/feature/coastal_app

and the regression test can be run with following command,

./rt.sh -l rt_coastal.conf -k -n coastal_florence_hsofs_atm2adc

to run the test you also need to stage the input files (there is no baseline yet). The adcprep is not part of this RT and the files generated by the adcprep is staged as a part of the input files. So, if you plan to test it in your side please let me know for the input file directory and required modifications in the rt.sh (this is required since I have no write access to UFS common data directory under platform, so I created one under my account). I am plaining to modify the RT once we are able to run the ADCIRC with CMEPS.

Once we push all those development to the UFS level, the input directory and baselines will be stored in common place for the all the supported platforms and also RTs.

uturuncoglu commented 1 year ago

@pvelissariou1 could you give me information about the file that I need to look at for the output. There are lots of fort* files and I could not be sure which one/s need to be to check the output. Is there any documentation that explains the all those files and their contents.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> The fort files to look are: fort.63.nc (Elevation Time Series at All Nodes in the Model Grid) fort.64.nc (Depth-averaged Velocity Time Series at All Nodes in the Model Grid)

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Mar 17, 2023 at 5:23 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 could you give me information about the file that I need to look at for the output. There are lots of fort* files and I could not be sure which one/s need to be to check the output. Is there any documentation that explains the all those files and their contents.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1474462705, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP62SN2DQZWKMSMGN6TW4TP5ZANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I have issue with those files. I tried to run the florence_hsofs.atm2adc on Orion via CoastalApp-testsuite (so the existing model - not my version) and nothing is written to those files while the model is running. Maybe I need to wait to finish the simulation and then the time dimension will be updated not sure but also file size not changing. So, probably I have issue with running model. As I understand those two files are created outside of the model run in the prep step. I think like this because if I delete those two files and run the model again, it fails during file initialization (write_output.F:1603). Anyway, at this point I have no baseline to compare (at least the files, I know the model that I am forcing with CDEPS uses different input). If you don't mind could you run the florence_hsofs.atm2adc case in your end (I think it is running on Hera without any issue) and copy the run directory to Cheyenne or Orion. So, if I could see the actual running case, maybe I could find what is going on wrong with it. Thanks.

uturuncoglu commented 1 year ago

Definitely there is an issue with the model or florence_hsofs.atm2adc configuration since PET0 seems finalized (ATMMESH) but ADCIRC PETs are still running on Orion. Again this is current version of coupled model not the modified and updated one.

uturuncoglu commented 1 year ago

@pvelissariou1 While we are solving issues related with the ADCIRC, i would like to start also bring SCHISM to the UFS model. It seems there are two version under CoastalApp: schism and schism-esmf. Which one do we need to port to UFS model? The schism might have CMake build but schism-esmf not. Again, if you give me more information about it, that would be great. Also, I'll try to use the CMake provided by the SCHISM but if it won't work I could create CMake interface for SCHISM under the UFS like I did in ADCIRC.

pvelissariou1 commented 1 year ago

Hi, Ufuk

It is schism. schism-esmf just contains the NUOPC cap for CoastalApp.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sat, Mar 18, 2023 at 2:02 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 While we are solving issues related with the ADCIRC, i would like to start also bring SCHISM to the UFS model. It seems there are two version under CoastalApp: schism and schism-esmf. Which one do we need to port to UFS model? The schism might have CMake build but schism-esmf not. Again, if you give me more information about it, that would be great. Also, I'll try to use the CMake provided by the SCHISM but if it won't work I could create CMake interface for SCHISM under the UFS like I did in ADCIRC.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1474753664, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP367DQ4ZYDZ5HFFPIDW4VMZNANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I am little bit confused because that one also has schism source code. https://github.com/schism-dev/schism-esmf/tree/master/src/schism do we need to checkout both of them under UFS?

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.**> From the schism-esmf we use only schism-esmf/Makefile schism-esmf/schism/ files in CoastalApp (the NUOPC cap for SCHISM). I don't know what is the purpose of the rest of the files in esmf-schism. I'll create an issue on SCHISM about this. We had long discussions in the past with the SCHISM developers; let me re-start this discussion again.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Sun, Mar 19, 2023 at 11:29 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I am little bit confused because that one also has schism source code. https://github.com/schism-dev/schism-esmf/tree/master/src/schism do we need to checkout both of them under UFS?

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1475606139, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4W2EP2WLC6S67LKRLW47MINANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I could not find any test with SCHISM under CoastalApp-testsuite. If you don't mind could you point me a test with SCHISM. So, I could compare the one under UFS. BTW, any progress of running florence_hsofs.atm2adc on Orion or any other platform successfully. In my side, I could not run.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> Have you cloned a recent version of the testsuite? git clone https://github.com/noaa-ocs-modeling/CoastalApp-testsuite.git -b develop All test cases with the "sch" substring in their filenames are the schism cases. I sent you a google invitation to see why the hsofs case is not working.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Mar 21, 2023 at 2:41 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I could not find any test with SCHISM under CoastalApp-testsuite. If you don't mind could you point me a test with SCHISM. So, I could compare the one under UFS. BTW, any progress of running florence_hsofs.atm2adc on Orion or any other platform successfully. In my side, I could not run.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1478482490, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7CJOWJP6UCKNZPIPLW5H77PANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Let me test it under Orion again with the latest version to see florence_hsofs.atm2adc will work or not.

uturuncoglu commented 1 year ago

@pvelissariou1 I tried with the develop. The florence_hsofs.atm2adc test fails with following error.

/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/ALLBIN_INSTALL/NEMS.x: error while loading shared libraries: libpnetcdf.so.4: cannot open shared object file: No such file or directory
/work/noaa/nems/tufuk/CoastalApp-testsuite/CoastalApp/ALLBIN_INSTALL/NEMS.x: error while loading shared libraries: libpnetcdf.so.4: cannot open shared object file: No such file or directory

setup job is fine.

uturuncoglu commented 1 year ago

I add following to model_run_slurm.job

module load cmake
module load intel/2018.4 impi/2018.4
module load hdf5/1.10.5-parallel pnetcdf/1.12.0 netcdf/4.7.2-parallel

and run it to see what happens. Module issue gone but I am getting

/var/spool/slurmd/job9500254/slurm_script: line 93: NEMS.x: command not found

error now.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> When you run the "run_all.sh" script you may pass all needed arguments to the script so it can load the needed modulefile and find the location of NEMS.x. I think I have a detailed explanation on the README.md file on how to run the script. Also you may run the script as "run_all.sh --help" to see all available options you may pass to the script. Please, see if the above works.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Mar 21, 2023 at 5:58 PM Ufuk Turunçoğlu @.***> wrote:

I add following to model_run_slurm.job

module load cmake module load intel/2018.4 impi/2018.4 module load hdf5/1.10.5-parallel pnetcdf/1.12.0 netcdf/4.7.2-parallel

and run it to see what happens. Module issue gone but I am getting

/var/spool/slurmd/job9500254/slurm_script: line 93: NEMS.x: command not found

error now.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1478701938, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZJS5XK6CCITMRHTMDW5IW7FANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. I fixed the issue and run the model. Still have issue with output. Model seems working fine but I don't have anything in the output files and time dimension is still 0. If you don't mind could you try to run the florence_hsofs.atm2adc case on Orion in your side and check the output. For example, I have no data in fort.64.nc and time dimension size is zero.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.> @Saeed Moghimi - NOAA Federal @.> I need to go into all cases and configure all models to write out their results. We had the testsuite cases to just run to make sure that all cases were running without errors. I'll try to do it as soon as possible.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Wed, Mar 22, 2023 at 12:26 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Okay. I fixed the issue and run the model. Still have issue with output. Model seems working fine but I don't have anything in the output files and time dimension is still

  1. If you don't mind could you try to run the florence_hsofs.atm2adc case on Orion in your side and check the output. For example, I have no data in fort.64.nc and time dimension size is zero.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1478936107, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4ERFWUSZEYRDOXAKDW5KEPFANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>