noaa-ocs-modeling / CoastalApp

CoastalApp will be deprecated. Please switch to https://github.com/oceanmodeling/ufs-coastal . CoastalApp is a NUOPC application implemented following UFS best practices to couple coastal ocean models and other domains (Sea Ice, Atmosphere, Wave, Inland Hydrology, ...)
https://github.com/oceanmodeling/ufs-coastal
Creative Commons Zero v1.0 Universal
13 stars 30 forks source link

CoastalApp compilation on Cheyenne fails #156

Open pvelissariou1 opened 1 year ago

pvelissariou1 commented 1 year ago
Ufuk Turuncoglu Ufuk Turuncoglu Jan 27, 2023, 1:16 PM (3 days ago)    

Ufuk Turuncoglu to Saeed, me, Daniel | to Saeed, me, Daniel to Saeed, me, Daniel

Ufuk Turuncoglu Jan 27, 2023, 1:16 PM (3 days ago) to Saeed, me, Daniel

Hi All,

I am following steps in

https://github.com/noaa-ocs-modeling/CoastalApp-testsuite

to run the tests. I am getting following error when I try to compile the model on Cheyenne. It seems that we have module files under modulefiles/ directory.

./build.sh --compiler intel --platform cheyenne --component "atmesh pahm adcirc ww3" -y envmodules_intel.cheyenne :: This environment is either not configured or not supported Exiting ...

Anyway, I just want to let you know. If you have no idea why it is not working I could try to fix it but I just want o ask you first. I also tried with gnu and it is not working too. Any idea? I could also try on Orion if you want.

Regards,

—ufuk

Ufuk Turuncoglu Jan 27, 2023, 1:17 PM (3 days ago) to Saeed, me, Daniel

Do i also need to follow the steps in compile-NEMS.x.txt (under CoastalApp directory)?

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov Jan 27, 2023, 1:29 PM (3 days ago) to Ufuk, Saeed, Daniel

Ufuk hi,

I haven't configured CoastalApp for "cheyenne" because I don't have access to this platform. What you can do is go into CoastalApp/modulefiles, copy the hera modulefiles to cheyenne and modify them according to cheyenne's module configuration. I think it will be better if we could have a chat (google meet) to discuss details. Are you free today?

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu Jan 27, 2023, 1:36 PM (3 days ago) to me, Saeed, Daniel

Hi,

I have just comment you the section in the module file (for Cheyenne, Intel) that causes issue. Now I am trying to build and see how it goes. If that won’t work I’ll try your suggestion and use Hera one as a base. I’ll let you know if I need help about it and then we could have a chat. Is there any other platform that is tested regularly such as Orion?

Thanks for your help,

—ufuk

Ufuk Turuncoglu Jan 27, 2023, 1:38 PM (3 days ago) to me, Saeed, Daniel

BTW, which Here module is the best one for Intel compiler. I could see multiple under module files/ directory. Thanks.

—ufuk

On Jan 27, 2023, at 12:29 PM, Panagiotis Velissariou - NOAA Affiliate [panagiotis.velissariou@noaa.gov](mailto:panagiotis.velissariou@noaa.gov) wrote:

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov Jan 27, 2023, 1:49 PM (3 days ago) to Ufuk

envmodules_intel.hera is the file currently used. Basically the code in all modulefiles is the same, only the modules that are loaded are different between platforms. Could you please load the module manually and send me what NETCDF and HDF environment variables are set?

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu Jan 27, 2023, 3:11 PM (3 days ago) to me

Hi,

I did following,

cc = mpicc cxx = mpicxx

To

if [[ -z "${CC}" ]]; then cc = mpicc else cc = ${CC} fi if [[ -z "${CXX}" ]]; then cxx = mpicxx else cxx = ${CXX} fi

Since the variables can be provided by the environment variables.

module purge module load cmake/3.22.0 module load ncarenv/1.3 module load intel/2022.1 module load mpt/2.25 module load ncarcompilers/0.5.0 module use /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1 module load hpc-mpt/2.25 export CC=mpicc export PCC=mpicc export CXX=mpicxx export PCXX=mpicxx export FC=mpif90 module load zlib/1.2.11 module load hdf5/1.10.6 module load netcdf/4.7.4 module load pio/2.5.7 module load esmf/8.3.0b09

When I run the script using following command,

./build.sh --compiler intel --platform cheyenne --component "atmesh pahm adcirc ww3" --tp parmetis -y

I am getting following error,

make[1]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/metis/build/Linux-x86_64' Makefile:11: *** missing separator. Stop. ERROR:: compileMetis (main): called from: functions_build the configuration of the ParMETIS library source failed tried to run the make command as: make config CC=mpicc CXX=mpicxx cc=mpicc cxx=mpicxx CFLAGS= CONFIG_FLAGS1=-DCMAKE_C_FLAGS='"-DIDXTYPEWIDTH=32 -DREALTYPEWIDTH=32"' gklib_path=/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/metis/GKlib metis_path=/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/metis prefix=/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/thirdparty_open/parmetis/del Exiting now …

I am still investigating. I’ll update you when I could compile the model then maybe you might want to push those changes to the main repository or I could create a branch and we could have a PR later.

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov Jan 27, 2023, 3:38 PM (3 days ago) to Ufuk

You don't need to modify any of these in the Makefile. All these are set by the build script. Check the file scripts/functions_build where there is a function compileMetis that defines all these variables and configures metis/parmetis properly. It should work in your case, if not please let me know the errors you are getting.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu Jan 29, 2023, 11:47 PM (12 hours ago) to me

Hi,

I past that issue but I am having another error as following,

-- Configuration on "cheyenne" waits verification. -- Found NetCDF: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include (found version "4.7.4") found components: C Fortran -- FindNetCDF defines targets: -- - NetCDF_VERSION [4.7.4] -- - NetCDF_PARALLEL [TRUE] -- - NetCDF_C_CONFIG_EXECUTABLE [/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/bin/nc-config] -- - NetCDF::NetCDF_C [STATIC] [Root: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4] Lib: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdf.a -- - NetCDF_Fortran_CONFIG_EXECUTABLE [/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/bin/nf-config] -- - NetCDF::NetCDF_Fortran [STATIC] [Root: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4] Lib: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdff.a CMake Error at cmake/check_netcdf.cmake:143 (message): The NetCDF library specified is not compatible with the specified compilers.It will not be enabled. Specify a different path or disable NetCDF.Ensure that you specify the same compilers to build PAHM as were used to build the NetCDF library.

Call Stack (most recent call first): CMakeLists.txt:238 (include)

-- Configuring incomplete, errors occurred!

and then

pahm_mod.F90(15): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [PAHM_MESH] USE PaHM_Mesh ------^ pahm_mod.F90(107): error #6404: This name does not have a type, and must have an explicit type. [NP] theData%numNd = np --------------------^ pahm_mod.F90(108): error #6404: This name does not have a type, and must have an explicit type. [NE] theData%numEl = ne --------------------^ pahm_mod.F90(122): error #6404: This name does not have a type, and must have an explicit type. [SLAM] theData%ndCoords((iCnt - 1) nDims + 1) = slam(iCnt) -------------------------------------------------^ pahm_mod.F90(123): error #6404: This name does not have a type, and must have an explicit type. [SFEA] theData%ndCoords((iCnt - 1) nDims + 2) = sfea(iCnt) -------------------------------------------------^ pahm_mod.F90(130): error #6404: This name does not have a type, and must have an explicit type. [NM] theData%elConnect((iCnt - 1) * numFaceEl + 1) = nm(iCnt, 1) ------------------------------------------------------^ pahm_mod.F90(136): error #6404: This name does not have a type, and must have an explicit type. [DP] theData%bathymetry = dp -------------------------^ compilation aborted for pahm_mod.F90 (code 1)

It seems that those parameters (i.e., np) is defined in PAHM/src/mesh.F90 and it needs to be compiled before pahm_mod.F90. Maybe it is because if netcdf issue, not sure.

At this point I put following to modulefiles/envmodules_intel.cheyenne (trying to use exact modules used for UFS),

module load cmake/3.22.0 module load ncarenv/1.3 module load intel/2022.1 module load mpt/2.25 module load ncarcompilers/0.5.0 module use /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1 module load hpc-mpt/2.25 export CC=mpicc export PCC=mpicc export CXX=mpicxx export PCXX=mpicxx export FC=mpif90 export F90=mpif90 export PFC=mpif90 export PF90=mpif90 module load zlib/1.2.11 module load hdf5/1.10.6 module load netcdf/4.7.4 module load pio/2.5.7 module load esmf/8.3.0b09

So, maybe there are some restriction in CostallApp side in terms of module versions. In my case, I am using exact modules and netcdf needs to be consistent with compiler. Maybe there is something in PHMS side. I’ll try to find the source. In the meantime, if you have any suggestion just let me know.

Thanks,

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov 9:00 AM (3 hours ago) to Ufuk

Hi Ufuk,

I have fixed these issues, there was a conflict with the schism model. Let me update the github repo and I'll let you know.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov 10:15 AM (2 hours ago) to Ufuk

Hi,

Please clone the develop-updates branch for the time being: git clone --recurse-submodules https://github.com/noaa-ocs-modeling/CoastalApp.git -b develop-updates

With this branch and your module file, it should compile on cheyenne. Please let me know.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu 10:43 AM (1 hour ago) Thanks for the quick fix. I’ll try and let you know.

Ufuk Turuncoglu 10:58 AM (1 hour ago) to me

Hi Again,

I am still getting same error with your branch,

-- - NetCDF::NetCDF_Fortran [STATIC] [Root: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4] Lib: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdff.a CMake Error at cmake/check_netcdf.cmake:143 (message): The NetCDF library specified is not compatible with the specified compilers.It will not be enabled. Specify a different path or disable NetCDF.Ensure that you specify the same compilers to build PAHM as were used to build the NetCDF library.

Call Stack (most recent call first): CMakeLists.txt:238 (include)

-- Configuring incomplete, errors occurred! See also "/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeOutput.log". Couldn't run "make" due to "cmake" errors. make[1]: Entering directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/nuopc'

mpif90 -c -g -traceback -fp-model precise -fPIC -O2 -fPIC -assume realloc_lhs -m64 -mcmodel=small -pthread -threads -qopenmp -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/esmf/8.3.0b09/mod -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/esmf/8.3.0b09/include -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_VERSION_STRING_GIT='v8.3.0b09' -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_MPIIO -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DSx86_64_small=1 -DESMF_OS_Linux=1 -DESMF_COMM=mpt -DESMF_DIR=/glade/work/jongkim/stacks/hash/hpc-stack-esmf/pkg/v8.3.0b09 -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -I. -I/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL -I/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/include -L/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib -L. -L/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL -L/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/lib -DWITHPETLISTS_on -DESMF_VERSION_MAJOR=8 pahm_mod.F90 pahm_mod.F90(15): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [PAHM_MESH] USE PaHM_Mesh ------^ pahm_mod.F90(107): error #6404: This name does not have a type, and must have an explicit type. [NP] theData%numNd = np --------------------^ pahm_mod.F90(108): error #6404: This name does not have a type, and must have an explicit type. [NE] theData%numEl = ne --------------------^ pahm_mod.F90(122): error #6404: This name does not have a type, and must have an explicit type. [SLAM] theData%ndCoords((iCnt - 1) nDims + 1) = slam(iCnt) -------------------------------------------------^ pahm_mod.F90(123): error #6404: This name does not have a type, and must have an explicit type. [SFEA] theData%ndCoords((iCnt - 1) nDims + 2) = sfea(iCnt) -------------------------------------------------^ pahm_mod.F90(130): error #6404: This name does not have a type, and must have an explicit type. [NM] theData%elConnect((iCnt - 1) * numFaceEl + 1) = nm(iCnt, 1) ------------------------------------------------------^ pahm_mod.F90(136): error #6404: This name does not have a type, and must have an explicit type. [DP] theData%bathymetry = dp -------------------------^ compilation aborted for pahm_mod.F90 (code 1) Makefile:48: recipe for target 'pahm_mod.o' failed make[1]: [pahm_mod.o] Error 1 make[1]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/nuopc' /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/NEMS/src/incmake/component_PAHM.mk:30: recipe for target '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/pahm.mk' failed make: [/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM_INSTALL/pahm.mk] Error 2

—ufuk

Panagiotis Velissariou - NOAA Affiliate panagiotis.velissariou@noaa.gov 11:56 AM (26 minutes ago) to Ufuk

Hi,

Unfortunately CoastalApp does not work with the hpc-stack static libraries. Can you use the OS supplied libraries for NetCDF/HDF5 and ESMF so all dependencies are resolved? Let me check on hera to see what compiles in CoastalApp using hpc-stack. For sure ADCIRC can not be compiled using hpc-stack. It is in the to-do list to add the hpc-stack functionality in CoastalApp.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu 12:00 PM (21 minutes ago) to me

Thanks for your help. Okay. I could try with others frowned by the system. In any case we need to fix this issue to make components work under UFS since they are using theses modules. If you could make it work under Hera just let me know. One more question, I think set of libraries are used by CostallApp that could be part of hpc-stack. Could you list them for me? I know parmetis is one of them, which I think you don’t need to compile every time with the app itself and could be part of the hpc-stack (or spack-stack in the near future). Right?

—ufuk

Panagiotis Velissariou - NOAA Affiliate 12:15 PM (7 minutes ago) to Ufuk

ParMetis can not be part of hpc-stack due to licensing issues. We had this discussion with NCEP and it was decided that ParMetis can not be part of the official NOAA modeling infrastructure due to its license. This is the reason I have the ParMetis functionality in CoastalApp. Now, WW3 is moving away from ParMetis (I don't know when this will be done) and SCHISM has the capability to use an externally compiled ParMetis or NOT.

As long as you compiled ParMetis once in CoastalApp (it is installed in CoastalApp/THIRDPARTY_INSTALL) you can rerun the build.sh scripts as: PARMETISHOME=FULL_PATH_TO_CoastalApp/THIRDPARTY_INSTALL ./build.sh "your options" (without the --tp parmetis option) and CoastalApp will pick-up the installed ParMetis. Just make sure that all libraries, including ParMetis, are compiled using the same compilers and mpi libraries.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: panagiotis.velissariou@noaa.gov

Ufuk Turuncoglu 12:14 PM (7 minutes ago) to me

I also tried to do this to see why netcdf is failing. I write the INFO variables from Cmake build (add message(INFO ${NetCDF3_LOG}) to PAHM/cmake/check_netcdf.cmake) and it seems that it is failing like following,

Change Dir: /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_95f2d/fast && gmake[1]: Entering directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp' /usr/bin/gmake -f CMakeFiles/cmTC_95f2d.dir/build.make CMakeFiles/cmTC_95f2d.dir/build gmake[2]: Entering directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp' Building Fortran object CMakeFiles/cmTC_95f2d.dir/netcdf3check.f90.o /glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/2022.1/mpi/mpif90 -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -extend-source 132 -heap-arrays 0 -fno-alias -sox -qno-opt-dynamic-align -O2 -debug minimal -nowarn -fp-model source -assume byterecl -mcmodel=medium -c /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/netcdf3check.f90 -o CMakeFiles/cmTC_95f2d.dir/netcdf3check.f90.o gmake[2]: No rule to make target '/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/hdf5/1.10.6/lib -lhdf5_hl -lhdf5 /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/zlib/1.2.11/lib -lz -ldl -lm -lnetcdf -lm', needed by 'cmTC_95f2d'. Stop. gmake[2]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp' Makefile:127: recipe for target 'cmTC_95f2d/fast' failed gmake[1]: [cmTC_95f2d/fast] Error 2 gmake[1]: Leaving directory '/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/PAHM/build/CMakeFiles/CMakeTmp'

Then, I copy the code that test netcdf3 and run the command manually.

/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/2022.1/mpi/mpif90 -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include -extend-source 132 -heap-arrays 0 -fno-alias -sox -qno-opt-dynamic-align -O2 -debug minimal -nowarn -fp-model source -assume byterecl -mcmodel=medium -c netcdf3check.f90

and it compiled code without any issue. It is also same netcdf4 one.So, I am not sure why this is not working under CMake build. I think it is not related with the hpc-stack modules. Anyway, I’ll try the find the source of the issue.

—ufuk

Ufuk Turuncoglu 12:20 PM (2 minutes ago) to me

Hi,

Okay. It is good to know the background about the parmetis. This will help when we start to move the components to UFS. Thanks.

—ufuk

uturuncoglu commented 1 year ago

@pvelissariou1 I might find the issue with error related with the netcdf library. It is cased by the following part in the cmake file,

  try_compile(NETCDF4_TEST
              "${CMAKE_BINARY_DIR}"
              SOURCES "${CMAKE_BINARY_DIR}/CMakeFiles/netcdf4check.f90"
              CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${NetCDF_INCLUDE_DIRS}"
              LINK_LIBRARIES "${NetCDF_LIBRARIES}" 
              LINK_LIBRARIES "${NetCDF_AdditionalLibs}"
              OUTPUT_VARIABLE NetCDF4_LOG)

The LINK_LIBRARIES part basically causes issue. It might be due to the used cmake version etc. under Cheyenne. If I remove them then it finds the target and compiler but fails due to the missing netcdf library but I think I could also provide them under CMAKE_FLAGS argument. Anyway, I'll try to fix it to see what happens.

pvelissariou1 commented 1 year ago

What version of cmake do you use? I usually load a cmake module that uses cmake version > 3.2

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Jan 30, 2023 at 12:35 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I might find the issue with error related with the netcdf library. It is cased by the following part in the cmake file,

try_compile(NETCDF4_TEST "${CMAKE_BINARY_DIR}" SOURCES "${CMAKE_BINARY_DIR}/CMakeFiles/netcdf4check.f90" CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${NetCDF_INCLUDE_DIRS}" LINK_LIBRARIES "${NetCDF_LIBRARIES}" LINK_LIBRARIES "${NetCDF_AdditionalLibs}" OUTPUT_VARIABLE NetCDF4_LOG)

The LINK_LIBRARIES part basically causes issue. It might be due to the used cmake version etc. under Cheyenne. If I remove them then it finds the target and compiler but fails due to the missing netcdf library but I think I could also provide them under CMAKE_FLAGS argument. Anyway, I'll try to fix it to see what happens.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1409128588, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP5ATR4YTDIGEBBACULWVACYDANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

It is cmake/3.22.0. The build is fine but it fails when linking in the try_compile command.

pvelissariou1 commented 1 year ago

Please load the NetCDF module and see what NETCDF environment variables are set. Also check the modulefiles/envmodulesintel.tacc to see how I set CoastalApp there. On TACC all environment variables start with TACC*

CoastalApp has been tested in many different systems with different NetCDF configurations and it works. Maybe you need to modify the module file slightly to fit Cheyenne's configuration.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Mon, Jan 30, 2023 at 12:40 PM Ufuk Turunçoğlu @.***> wrote:

It is cmake/3.22.0. The build is fine but it fails when linking in the try_compile command.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1409139116, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPYJVL3FFTRMXKUYDADWVADJVANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I think I fixed the issue with PAHM build. The issue was ordering of the netcdf libraries in the link stage. The build requires to add first netcdf fortran libraries and then netcdf c ones. Otherwise, the build complains about missing netcdf functions. I updated cmake/Modules/FindNetCDF.cmake file and also add list( REVERSE _search_components ) to change the order of the language components. Maybe this could be done in a more elegant way but since the script first checks C and then others this was the easiest solution. Anyway, if you need more information I could create a branch and push the changes to there. Or we could implement another way.

Anyway, that allowed me to pass the issue but then I hit to another one from ADCIRC. It was complaining about finding netcdf library but I fixed it by setting export NETCDF_DIR=$NETCDF_ROOTin my Cheyyene module file. Then, I got following error from LibXdmf libraries.

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
XDMF_LibXdmf
    linked by target "cmTC_73a97" in directory /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/ADCIRC/build/CMakeFiles/CMakeTmp
XDMF_LibXdmfCore
    linked by target "cmTC_73a97" in directory /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/ADCIRC/build/CMakeFiles/CMakeTmp
XDMF_LibXdmfUtils
    linked by target "cmTC_73a97" in directory /glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/ADCIRC/build/CMakeFiles/CMakeTmp

CMake Error at cmake/xdmf_check.cmake:40 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  CMakeLists.txt:105 (include)

In the following documentation, it says it is required to write data in Xdmf format.

https://wiki.adcirc.org/Compiling

I am not sure it is required or not to run the tests. Anyway, do I need to install it or checkout somewhere that ADCIRC could find and install it for me?

uturuncoglu commented 1 year ago

Anyway, I think I fixed that one too. I edit ADCIRC/work/makefile and again change the order of the link libraries for netcdf from -lnetcdf -lnetcdff to -lnetcdff -lnetcdf. Then, it could complete the build without any issue. This needs to be done for every occurrence of -lnetcdf -lnetcdff. I was also getting error related with the HDF library since netcdf is build with HDF and make some changes in the makefile to add them to the link stage too. I also modify the ADCIRC/work/makefile to use nf-config --flibs and nc-config --libs to get libraries related with netcdf and it seems it is working. I have issue like /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib/libnetcdf.a(libdispatch_la-dparallel.o): undefined reference to symbol 'MPI_Info_f2c'but I'll investigate it too.

uturuncoglu commented 1 year ago

This could be related with using fort rather than mpif90 in ADCIRC build and I think can be fixed easily. Anyway, I'll keep you updated.

pvelissariou1 commented 1 year ago

Thank you Ufuk, I'll take a look at these as soon as possible. Thi might be an issue for other models as well.

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 2, 2023 at 12:19 AM Ufuk Turunçoğlu @.***> wrote:

This could be related with using fort rather than mpif90 in ADCIRC build and I think can be fixed easily. Anyway, I'll keep you updated.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1413211730, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP5N7LX5DXLQKGRTOPTWVNGXHANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I was looking into MPI error in the linking step of ADCIRC.

patch_la-dparallel.o): undefined reference to symbol 'MPI_Info_f2c'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: /glade/u/apps/ch/opt/mpt/2.25/lib/libmpi.so: error adding symbols: DSO missing from command line

This seems related with the used compiler for ADCIRC and my build is using ifort rather than MPI compiler. It seems that there is no configure.nems.cheyenne.intel file under conf/ directory. Do we need to create one? I could easily use configure.nems.hera.intel or configure.nems.orion.intel as a reference to create it. Maybe because of the missing file it tries to run with default compiler.

uturuncoglu commented 1 year ago

If I modify ADCIRC/work/cmplrflags.mk and change PPFC from ifort to mpif90 under Intel section, the ADCIRC passes the MPI issue (there is no need to create configure.nems.cheyenne.intel) but then fails with following,

mpif90 -c -g -traceback -fp-model precise   -fPIC -O2 -fPIC -assume realloc_lhs -m64 -mcmodel=small -pthread -threads  -qopenmp -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/esmf/8.3.0b09/mod -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/esmf/8.3.0b09/include -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include  -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_VERSION_STRING_GIT='v8.3.0b09' -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_MPIIO -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DSx86_64_small=1 -DESMF_OS_Linux=1 -DESMF_COMM=mpt -DESMF_DIR=/glade/work/jongkim/stacks/hash/hpc-stack-esmf/pkg/v8.3.0b09 -I. -I/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/ADCIRC/work/odircp -I/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/ADCIRC/prep -I/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/include    -L. -L/glade/scratch/turuncu/COASTAL_APP/CoastalApp-testsuite/CoastalApp/ADCIRC  -L/glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/intel-2022.1/mpt-2.25/netcdf/4.7.4/lib -DWITHPETLISTS_on  -DESMF_VERSION_MAJOR=8 couple2swan_modif.F90
couple2swan_modif.F90(17): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [SIZES]
      USE SIZES,  ONLY: SZ
----------^
couple2swan_modif.F90(18): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [WRITE_OUTPUT]
      USE WRITE_OUTPUT, ONLY : terminate
----------^
couple2swan_modif.F90(19): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [GLOBAL]

So, maybe dependency needs to be define in ADCIRC/thirdparty/nuopc/Makefile. The sizes.F is under ADCIRC/src.

uturuncoglu commented 1 year ago

I am not sure how CoastalApp builds on other systems such as Hera without any issue. Maybe I am doing something totally wrong in here but in any case, if we want to fork UFS and port these components to there, the build needs to work with those modules. We could still follow the same approach and wrap each component with CMake build interfaces in UFS but I think it is better to use their own build system. BTW, some components have both cmake and make builds system like ADCIRC. Am I right? I am asking because UFS uses CMake for component builds.

pvelissariou1 commented 1 year ago

Ufuk hi, I have updated the CoastalApp README file and now includes detailed instructions on how to compile it. This might help. Also make sure when rebuild the application, to clean first the targets. Monday, I would like to have a Google meeting with you, I need to ask you a few questions as well. The latest and up to date branch is the develop branch.

On Saturday, February 4, 2023, Ufuk Turunçoğlu @.***> wrote:

I am not sure how CoastalApp builds on other systems such as Hera without any issue. Maybe I am doing something totally wrong in here but in any case, if we want to fork UFS and port these components to there, the build needs to work with those modules. We could still follow the same approach and wrap each component with CMake build interfaces in UFS but I think it is better to use their own build system. BTW, some components have both cmake and make builds system like ADCIRC. Am I right? I am asking because UFS uses CMake for component builds.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci6.googleusercontent.com/proxy/oLn-hAzD6G76W5B2vdltP1DR8RhHmFXU-LeiZkX_iNlpGwT6SZR49PvZYcPsfVcioCVsFoPULR5QWPcIDi38Ii0uGVsDMKphcTkqEdFCXIQp8uFiakOY-S15UYUFyP9kO11r-sOFqX6qL-6w29yEUtIlfy6dphBHyGQbjILl0lPvXwSjVk7BqRvuwK52WwQrgUsIvmQFMlhvbrJrScNJ1D1u9jkR0EV0xZ9-lDJdQOJ6MrqSqQ=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TPZCJ3YVKLKF3RLXMO3WVXZUZA5CNFSM6AAAAAAULPOB22WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSUODEA2.gif>Message ID: @.***>

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

pvelissariou1 commented 1 year ago

An update: 1) Clone CoastalApp develop branch 2) Compile CoastalApp on Orion using the orion default module 3) Compile CoastalApp on Orion using hpc-stack

uturuncoglu commented 1 year ago

@pvelissariou1 I confirmed that I could able to build CoastalApp on Orion and run the test cases without any issue with out-of-box modules. As a next step, I'll try to do same with the modules files used by UFS develop (hpc-stack). BTW, where all those run directories. It might be good to check the results visually to be sure that they run without any issue. I saw them in the queue but I am not sure they run successfully.

When I run the run_all.sh, I had issue as following,

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

Since I am not in coastal group, I need to change the job script using following command,

find ./ -name "model_setup.job" -type f -exec sed -i 's/--account=coastal/--account=nems/g' {} \;
find ./ -name "model_run.job" -type f -exec sed -i 's/--account=coastal/--account=nems/g' {} \;

Then, I got another one like,

sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

So, I had to add following to the job scripts

#SBATCH --qos=batch
#SBATCH --partition=orion

using following commands,

find ./ -name "model_setup.job" -type f -exec sed -i '2i #SBATCH --qos=batch\n#SBATCH --partition=orion' {} \;
find ./ -name "model_run.job" -type f -exec sed -i '2i #SBATCH --qos=batch\n#SBATCH --partition=orion' {} \;

So, maybe those information could come from a configuration file etc. Anyway, I think once we port the components to the UFS, I think it would be nice to port those tests to the UFS RT system and all these things are handled by RT system without any additional development.

uturuncoglu commented 1 year ago

@pvelissariou1 I think ADCIRC writes the data separately for each PET and needs to be combined together to check the results. Anyway, is there any post-processing tool that I could use to check the results?

pvelissariou1 commented 1 year ago

Thank you Ufuk. I am in the process of updating the testsuite scripts to be able to run in various HPC environments. Currently, the testsuite runs on hera. I'll keep you posted (by tomorrow I believe).

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 9, 2023 at 12:29 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think ADCIRC writes the data separately for each PET and needs to be combined together to check the results. Anyway, is there any post-processing tool that I could use to check the results?

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1424636439, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP2ADGKIMUNST4R267LWWUZPDANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I also tried with hpc-stack and I got similar error that I had on Cheyenne. It is related to the netcdf again.

--   - NetCDF::NetCDF_Fortran [STATIC] [Root: /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4] Lib: /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/lib/libnetcdff.a 
CMake Error at cmake/check_netcdf.cmake:143 (message):
  The NetCDF library specified is not compatible with the specified
  compilers.It will not be enabled.  Specify a different path or disable
  NetCDF.Ensure that you specify the same compilers to build PAHM as were
  used to build the NetCDF library.

Call Stack (most recent call first):
  CMakeLists.txt:238 (include)

Probably, we need to work on build system to make components happy with the hpc-stack before starting to move them under UFS. For your reference, here are my modules for modulefiles/envmodules_intel.orion

module load cmake/3.22.1
module use /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/modulefiles/stack
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load zlib/1.2.11
module load hdf5/1.10.6
module load netcdf/4.7.4
module load pio/2.5.7
module load esmf/8.3.0b09
uturuncoglu commented 1 year ago

@pvelissariou1 Do you want me to work on it? I could create fork for component and work on branches that fix the issue with the build.

pvelissariou1 commented 1 year ago

Yes, I'll work on this as well. I hope I will have something to share with you at our next meeting. Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 9, 2023 at 12:44 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I also tried with hpc-stack and I got similar error that I got without Cheyenne. It is related with the netcdf again.

-- - NetCDF::NetCDF_Fortran [STATIC] [Root: /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4] Lib: /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/lib/libnetcdff.a CMake Error at cmake/check_netcdf.cmake:143 (message): The NetCDF library specified is not compatible with the specified compilers.It will not be enabled. Specify a different path or disable NetCDF.Ensure that you specify the same compilers to build PAHM as were used to build the NetCDF library.

Call Stack (most recent call first): CMakeLists.txt:238 (include)

Probably, we need to work on build system to make components happy with the hpc-stack before starting to move them under UFS. For your reference, here are my modules for modulefiles/envmodules_intel.orion

module load cmake/3.22.1 module use /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1.2 module load hpc-impi/2022.1.2 module load zlib/1.2.11 module load hdf5/1.10.6 module load netcdf/4.7.4 module load pio/2.5.7 module load esmf/8.3.0b09 ``

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1424656212, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP45BONXLSLAJRIT6VTWWU3HHANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

Yes, that will be great. Just make a fork based on the develop branch. What about if we make a feature/ncar branch in CoastalApp and work on this?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 9, 2023 at 12:46 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Do you want me to work on it? I could create fork for component and work on branches that fix the issue with the build.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1424658351, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPYGZHHXKXZPMSEPZ6TWWU3PBANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

Sure. feature/ncar will be fine.

pvelissariou1 commented 1 year ago

I just created the future/ncar branch in CoastalApp (based on develop).

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 9, 2023 at 12:55 PM Ufuk Turunçoğlu @.***> wrote:

Sure. feature/ncar will be fine.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1424668677, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPZTDIBJKRSJP4RCI33WWU4TNANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

The following test are still running, is it expected,

           8925869     orion fl_wav_m    tufuk  R    2:22:37     15 Orion-01-[03,43],Orion-02-[27,40,51],Orion-06-[61-62],Orion-08-[08,47-48,72],Orion-09-[03,06],Orion-15-[06-07]
           8925867     orion fl_atm2w    tufuk  R    2:22:41     15 Orion-11-[32-46]
           8925871     orion fl_wav_n    tufuk  R    2:15:30     15 Orion-17-[38-46],Orion-23-[69-72],Orion-24-[01-02]

If I remember correctly they were very cheap tests but maybe I am wrong.

pvelissariou1 commented 1 year ago

Maybe a process waits another process. Did you check the log files in the run directory?

On Thursday, February 9, 2023, Ufuk Turunçoğlu @.***> wrote:

The following test are still running, is it expected,

8925869 orion fl_wav_m tufuk R 2:22:37 15 Orion-01-[03,43],Orion-02-[27,40,51],Orion-06-[61-62],Orion-08-[08,47-48,72],Orion-09-[03,06],Orion-15-[06-07] 8925867 orion fl_atm2w tufuk R 2:22:41 15 Orion-11-[32-46] 8925871 orion fl_wav_n tufuk R 2:15:30 15 Orion-17-[38-46],Orion-23-[69-72],Orion-24-[01-02]

If I remember correctly they were very cheap tests but maybe I am wrong.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci5.googleusercontent.com/proxy/LFeOjEjYtR-20cgSRaxX0oWiw8i_TIYOfvjw8j8st-IvdVaim_WolohPt9R5F_pHxoMgefbkba-vYZD92GtiPFqlPfNLkjVdzXtC6lDo8hXoL-eUUizfLWASehUjKJ6fTHBS8TEz_uB0nrL2841NgURFBmdNWnJcyC0DOUv-afLo9Me5H8radMGvjnM98Ltc_E1IpaY6J_sPMab536-iytoDAuxLGdEwP0WgA_toweHH5aWOjA=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TPZOY3MDK4SNEFXB2E3WWVJAHA5CNFSM6AAAAAAULPOB22WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSU5SZVO.gif>Message ID: @.***>

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

uturuncoglu commented 1 year ago

I can't see anything in the log. They could hang. Anyway, I'll keep monitor them. They are running more than 2-hours. Is it expected?

uturuncoglu commented 1 year ago

@pvelissariou1 I am creating forks for the component that I need to modify in my personal account due to the restriction issue. I think this is more flexible way to continue to the development. I have already fixed PAHM (https://github.com/uturuncoglu/PaHM and feature/ncar branch - just remove C netcdf library which is not required by the model) and I'll go one by one.

uturuncoglu commented 1 year ago

@pvelissariou1 It turns out I have still issue with PAHM build when it is triggered by the app.

pvelissariou1 commented 1 year ago

Ok, thanks. Let me finish with the test suite and I will check on PaHM. ADCIRC uses the same library setup as PaHM.

On Thursday, February 9, 2023, Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 It turns out I have still issue with PAHM build when it is triggered by the app.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.< https://ci5.googleusercontent.com/proxy/jgXiYmkk4VV-JyklsfsFdcCgcV4zvDsHOay-1bJP2Wjf31jXmOLOehKHM8VbVL1kdIlXp19XX7X1JWeBy6pDASFl-nV4gVixyWXliNe36lK0Gzhz6tb-7YBPVGjXGWqmpHKh09A9tkS5Zp3DPeMNUL5LnE3IBVeq2wNM__5UWiJfemgXWNIECwYMEmTVmJJryx12FlGDg2muIQvG53rIl3S-pHcHWPjNrVlRiMGDPDRWytBWuw=s0-d-e1-ft#https://github.com/notifications/beacon/APC7TP2IBFJQ7A2AUIBABCDWWVVR3A5CNFSM6AAAAAAULPOB22WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSU52LBA.gif>Message ID: @.***>

-- Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

uturuncoglu commented 1 year ago

@pvelissariou1 i have good news. I could able to build ADCIRC under UFS on Cheyenne. As I told you before in the call, I went with new CMake interface approach which is also used by some other UFS components. I think this is the best way to support both UFS and also other platforms outside of UFS at this point. We could discuss it more in the Kick-off meeting.

I think same version will also build on other platforms without any issue but I did not tested yet.

Here is my fork,

https://github.com/uturuncoglu/ufs-weather-model/tree/feature/coastal_app

At this point, I am only building only ADCIRC (no adcprep etc.) with external Metis (build with spack package manager) and this is preliminary work. I will modify the CMake interface (https://github.com/uturuncoglu/ufs-weather-model/blob/feature/coastal_app/ADCIRC-interface/CMakeLists.txt) to build with its internal Metis. So, there won't be any Metis dependency at least for ADCIRC. I also need to look at CPP options that is used by CoastalApp and be sure that I am using same ones.

I am plaining to go step by step and bring each component individually and test them. Once they works individually and produce similar answer with the CoastalApp. Then, we could couple them.

At this point my questions are;

uturuncoglu commented 1 year ago

@pvelissariou1 I think Metis is not used by adcirc but its prep. So, is this prep a tool to create some input file? Is it an executable? Anyway, more information with other components of ADCIRC will help to define which needs to be build under UFS.

uturuncoglu commented 1 year ago

@pvelissariou1 I added utilities and also tested with GNU on Cheyenne. No issue at this point, I'll add other components of build one by one.

uturuncoglu commented 1 year ago

@pvelissariou1 I think CMake build for ADCIRC does not have capability to build with NUOPC cap. Am I right? I think you were planing to work on it since i am not seeing any source file like ADCIRC/thirdparty/nuopc/adc_cap.F90 in the cmake/ folder.

pvelissariou1 commented 1 year ago

Yes, NUOPC Cap cannot be compiled using CMake. I'll be working on that as well.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 16, 2023 at 1:34 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think CMake build for ADCIRC does not have capability to build with NUOPC cap. Am I right? I think you were planing to work on it since i am not seeing any source file like ADCIRC/thirdparty/nuopc/adc_cap.F90 in the cmake/ folder.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1432645071, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4PURO2SWBD6EDMID3WXXJ7JANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

Need to collaborate with the developers of all models on compiling NUOPC using CMake

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 16, 2023 at 9:26 AM Panagiotis Velissariou - NOAA Affiliate < @.***> wrote:

Yes, NUOPC Cap cannot be compiled using CMake. I'll be working on that as well.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 16, 2023 at 1:34 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think CMake build for ADCIRC does not have capability to build with NUOPC cap. Am I right? I think you were planing to work on it since i am not seeing any source file like ADCIRC/thirdparty/nuopc/adc_cap.F90 in the cmake/ folder.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1432645071, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4PURO2SWBD6EDMID3WXXJ7JANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 BTW, what is the best way to test ADCIRC standalone. Which test found in https://github.com/noaa-ocs-modeling/CoastalApp-testsuite is the most suitable for it?

pvelissariou1 commented 1 year ago

florence_shinnecock.atm2adc

You might need to modify run/model_setup.job and run/model_run.job for the account, batch fields

Takis

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 16, 2023 at 10:36 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 BTW, what is the best way to test ADCIRC standalone. Which test found in https://github.com/noaa-ocs-modeling/CoastalApp-testsuite is the most suitable for it?

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1433373844, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4JNISMATOGTV6JTSDWXZJRTANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Thanks. I am plaining to define it as RT under UFS (I could create CoastalApp specific rt.conf file such as rt_coastal.conf. Then, we could move the test that you want to regularly test to rt.conf once this development goes to UFS authoritative repository. It would be nice to use existing infrastructure for it. So, we could test them quickly along with the development. Does it make sense to you?

pvelissariou1 commented 1 year ago

Yes, I agree. Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Thu, Feb 16, 2023 at 11:00 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 Thanks. I am plaining to define it as RT under UFS (I could create CoastalApp specific rt.conf file such as rt_coastal.conf. Then, we could move the test that you want to regularly test to rt.conf once this development goes to UFS authoritative repository. It would be nice to use existing infrastructure for it. So, we could test them quickly along with the development. Does it make sense to you?

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1433409653, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7SCDEKHM42FR2ADX3WXZMI3ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I could able to build parallel version of ADCIRC with coupling support. At this point I am testing florence_shinnecock.atm2adc case and trying to port ATMMESH to CDEPs to provide u10, v10 and surface pressure. I have already created a mode under CDEPS. Now I need to create ESMF Mesh file for wind_atm_fin_ch_time_vec.nc. It seems this data on nodes and I think it is coming from WW3. Is there any tool that could create ESMF mesh file for this dataset? I could see some code in the https://github.com/noaa-ocs-modeling/ATMESH/blob/a337c93b8ba5b97f9102c40778e9c8154175e004/atmesh.F90#L382 that write ESMF mesh (atmesh_mesh.nc) but I am not seeing that file anywhere.

At this point, my aim is to run florence_shinnecock.atm2adc case with ADCIRC+CDEPS and I am creating run directory manually since current configuration files that are found in the CoastalApp-testsuite repo are compatible with old version and needs to be ported to current UFS. After, having successful run, I'll try to run the same case with CMEPS and then make this case as a RT under UFS.

uturuncoglu commented 1 year ago

Maybe those files are in comm folder. BTW, there are some configuration options in atm_namelist.rc and I think those are used by ATMMESH. Right? At this point, I am not sure about extra capability that is provided by ATMMESH but not by CDEPS. Is there any documentation related with ATMMESH? Do we need to port those capabilities to CDEPS maybe under special data mode.

uturuncoglu commented 1 year ago

@pvelissariou1 Maybe it would be nice to go with florence_hsofs.atm2adc at this point. I think its input files (incl. ESMF mesh file) are in comm directory.

uturuncoglu commented 1 year ago

@pvelissariou1 okay. same issue also for florence_hsofs.atm2adc case. The ADCIRC runs under UFS but it fails when it tries to get data from CDEPS. I need to create ESMF mesh file for Florence_HWRF_HRRR_HSOFS.nc data set. It seems it is written as 1d array on nodes. I need to have original dataset (2d version, not on nodes but in lat,lon). Then, I could create SCRIP grid definition file and ESMF Mesh file using coordinates found in the original dataset and use it. At this point, I am not sure which grid type is used by the original dataset but both regular lat-lon and curvilinear grid works for me. CDEPS expect to have original dataset as well as ESMF mesh file to provide required fields to other components. We might need to do the same for the other model configurations that we want to port to UFS.

Except those issue here are my other questions:

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> The ESMF mesh data are created using the subroutines "construct_meshdata_from_netcdf" and "create_parallel_esmf_mesh_from_meshdata" found in atmesh_mod.F90. Please keep in mind that both atmesh and ww3data caps were designed to be used with ADCIRC and WW3. Now, SCHISM uses similar mesh definition as ADCIRC therefore, these caps work with SCHISM as well.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Feb 17, 2023 at 2:47 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I could able to build parallel version of ADCIRC with coupling support. At this point I am testing florence_shinnecock.atm2adc case and trying to port ATMMESH to CDEPs to provide u10, v10 and surface pressure. I have already created a mode under CDEPS. Now I need to create ESMF Mesh file for wind_atm_fin_ch_time_vec.nc. It seems this data on nodes and I think it is coming from WW3. Is there any tool that could create ESMF mesh file for this dataset? I could see some code in the https://github.com/noaa-ocs-modeling/ATMESH/blob/a337c93b8ba5b97f9102c40778e9c8154175e004/atmesh.F90#L382 that write ESMF mesh (atmesh_mesh.nc) but I am not seeing that file anywhere.

At this point, my aim is to run florence_shinnecock.atm2adc case with ADCIRC+CDEPS and I am creating run directory manually since current configuration files that are found in the CoastalApp-testsuite repo are compatible with old version and needs to be ported to current UFS. After, having successful run, I'll try to run the same case with CMEPS and then make this case as a RT under UFS.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1435248964, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4KME4IT5PSCNVZD4DWX7PU7ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> The options in "config.rc" and not in "atm_namelist.rc" are specific to ATMESH. "atm_namelist.rc" should (in our case) be a link (need to fix this, or delete this file) to "model_configure" that contains the values of model configured variables. Example, start_time, simulation_length, etc, ... The atm_namelist.rc is not used anywhere in CoastalApp, unless an atmospheric model within UFS is present.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Feb 17, 2023 at 2:57 PM Ufuk Turunçoğlu @.***> wrote:

Maybe those files are in comm folder. BTW, there are some configuration options in atm_namelist.rc and I think those are used by ATMMESH. Right? At this point, I am not sure about extra capability that is provided by ATMMESH but not by CDEPS. Is there any documentation related with ATMMESH? Do we need to port those capabilities to CDEPS maybe under special data mode.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1435258625, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP7SXANWRAADOXA7FD3WX7QZRANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> The only functionality of ATMESH is to read already created atmospheric data on an ADCIRC/SCHISM mesh, create the ESMF_mesh and pass the data to relevant models. So, CDEPS is much more general and should, I believe, replace both ATMESH/WW3DATA.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Fri, Feb 17, 2023 at 2:47 PM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I could able to build parallel version of ADCIRC with coupling support. At this point I am testing florence_shinnecock.atm2adc case and trying to port ATMMESH to CDEPs to provide u10, v10 and surface pressure. I have already created a mode under CDEPS. Now I need to create ESMF Mesh file for wind_atm_fin_ch_time_vec.nc. It seems this data on nodes and I think it is coming from WW3. Is there any tool that could create ESMF mesh file for this dataset? I could see some code in the https://github.com/noaa-ocs-modeling/ATMESH/blob/a337c93b8ba5b97f9102c40778e9c8154175e004/atmesh.F90#L382 that write ESMF mesh (atmesh_mesh.nc) but I am not seeing that file anywhere.

At this point, my aim is to run florence_shinnecock.atm2adc case with ADCIRC+CDEPS and I am creating run directory manually since current configuration files that are found in the CoastalApp-testsuite repo are compatible with old version and needs to be ported to current UFS. After, having successful run, I'll try to run the same case with CMEPS and then make this case as a RT under UFS.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1435248964, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP4KME4IT5PSCNVZD4DWX7PU7ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

I could see the call in atmesh_mod.F90 that creates ESM Mesh file but at this point there is no way to write mesh in netcdf format under ESMF (I'll check it with Bob again.). So, we need to create ESMF mesh with offline tool to provide it to the CDEPS. Of course we could try to port that code to CDEPS but it is application specific. So, it would be better to create that mesh outside of CDEPS and provide it as input.

pvelissariou1 commented 1 year ago

Ok, I see. We don't have such a tool in place. Yes, we cannot be application specific, we need a general tool. If this general tool is to be developed from scratch (source code development), can its code be part of CDEPS, or has to be external to CDEPS?

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Feb 21, 2023 at 11:02 AM Ufuk Turunçoğlu @.***> wrote:

I could see the call in atmesh_mod.F90 https://github.com/noaa-ocs-modeling/ATMESH/blob/a337c93b8ba5b97f9102c40778e9c8154175e004/atmesh_mod.F90#L416 that creates ESM Mesh file but at this point there is no way to write mesh in netcdf format under ESMF (I'll check it with Bob again.). So, we need to create ESMF mesh with offline tool to provide it to the CDEPS. Of course we could try to port that code to CDEPS but it is application specific. So, it would be better to create that mesh outside of CDEPS and provide it as input.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1438815180, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TP742KQFHU3CEUXVPNTWYTYI7ANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 I think this needs to be external to CDEPS something like a offline tool. It will get mesh definition that is used by ADCIRC and other model components and convert it to the ESMf mesh file. BTW, if the atmospheric input such as surface pressure, u and v is in a different grid (such as regular lat,lon etc.) then maybe we don't need to create such tool since we could perform interpolation on the fly. What is the source of those forcing? Are they in a grid different than the ADCIRC mesh? If so, could you point me one of those files.

pvelissariou1 commented 1 year ago

@Ufuk Turuncoglu @.***> In general, the atmospheric fields (model outputs) are not defined in an ocean model specific grid. Usually, we interpolate from atm->ocean mesh (offline in our case) and supply the interpolated data to the ocean model. So, this means that the interpolation can be done within CDEPS as the data can be passed to more than one ocean/wave model (that may have different mesh definitions). Yes, basically we don't need (and we don't want) an external tool to create the NetCDF ESMF_mesh files. Shortly, I'll send you links to various atm. grid definitions used routinely.

Panagiotis Velissariou, Ph.D., P.E. UCAR Scientist National Ocean and Atmospheric Administration National Ocean Service Office of Coast Survey CSDL/CMMB Physical Scientist - Project Lead cell: (205) 227-9141 email: @.***

On Tue, Feb 21, 2023 at 11:25 AM Ufuk Turunçoğlu @.***> wrote:

@pvelissariou1 https://github.com/pvelissariou1 I think this needs to be external to CDEPS something like a offline tool. It will get mesh definition that is used by ADCIRC and other model components and convert it to the ESMf mesh file. BTW, if the atmospheric input such as surface pressure, u and v is in a different grid (such as regular lat,lon etc.) then maybe we don't need to create such tool since we could due interpolation on the fly. What is the source of those forcing? Are they in a grid different than the ADCIRC mesh? If so, could you point me one of those files.

— Reply to this email directly, view it on GitHub https://github.com/noaa-ocs-modeling/CoastalApp/issues/156#issuecomment-1438846647, or unsubscribe https://github.com/notifications/unsubscribe-auth/APC7TPY4JSJKAU6FA6Z55ETWYT3ADANCNFSM6AAAAAAULPOB2Y . You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 1 year ago

@pvelissariou1 Okay. That is great. If I could create mesh file for the original data file then CDEPS will handle rest.