search

oceanmodeling / ufs-weather-model

This repo is forked from ufs-weather-model, and contains the model code and external links needed to build the UFS coastal model executable and model components, including the ROMS, FVCOM, ADCIRC and SCHISM plus WaveWatch III model components.
https://github.com/oceanmodeling/ufs-coastal-app
Other
2 stars 3 forks source link

ROMS Integration #9

Open uturuncoglu opened 10 months ago

uturuncoglu commented 10 months ago

@hga007 @saeed-moghimi-noaa @pvelissariou1 I am creating new issue for ROMS integration. So, we could keep communicating about integration issues that we might have.

saeed-moghimi-noaa commented 10 months ago

Thanks @uturuncoglu

hga007 commented 10 months ago

Sounds good. Thank you! Although we have not publicly announced it, we have ROMS available on GitHub. We are still working on a few things before it is released, and we hope we are done this week (see https://github.com/myroms).

saeed-moghimi-noaa commented 10 months ago

Thanks @hga007, very timely!

pvelissariou1 commented 10 months ago

@hga007 , Dear Hernan as we are trying to incorporate ROMS into UFS-Coastal, we would like to have a couple of tests cases in place for ROMS. If possible, could you supply simple test cases for ROMS such as: (a) ROMS forced by atmospheric data through the data component (ATM->ROMS, may be a 2D case), and (b) a ROMS and CICE coupled case. It will be very helpful to have these test cases, to test thing out during our ROMS/UFS-Coastal works. Thank you Takis

hga007 commented 10 months ago

Many test cases are available in the roms_test repository (https://github.com/myroms/roms_test). I suggested to Ufuk a few months ago to use the Hurricane IRENE. It is fully set up to run with data (Forward sub-directory), coupled to WRF (Coupled sub-directory), coupled with ocean 4D-Var data assimilation (Coupled_RBL4DVAR). It uses ROMS native 4D-Var algorithms, and I will probably add it to the ROMS-JEDI interface to run with FV3 in the future. During the coupling development, using its native NUOCP-based coupling infrastructure, I tested forcing ROMS with its original forcing NetCDF files and coupled DATA model component in applications along the US west and east coasts and the Indian Ocean.

We don't have 2D shallow-water applications for coupling. They don't make much sense to me because we need to test the essential aspects of the surface boundary layer between the atmosphere and the ocean. It is not a good test for developers. Other versions of ROMS out there use CICE6. I wrote a NUOPC cap for CICE6 in ROMS native framework. However, it hasn't been a top priority in my research to include sea ice applications. We did propose to include sea ice and adjoint-based algorithms, but it was not funded.

uturuncoglu commented 10 months ago

@hga007 3d case will be fine but I prefer to have a case very simple just ROMs forced with DATM. Just for clarification, we are plaining to use CDEPS to force the ROMS. I know ROMS has its own data component capability and I think that we could use the same files under CDEPS. The only think is that configuring ROMS to coupled with DATM but the CDEPS one. Also, uff-weather model has CICE6 cap and if we want to couple ROMS with CICE, we need to use that one. I am not sure what is the main difference between CICE cap under ROMs and the one that is already found in CICE code. That needs to be explored and if ROMS cap has some extra features we need to bring them to CICE6 one. Of course that is not wfor now. We just need to start from simple DATM+ROMS configuration as a initial step.

pvelissariou1 commented 10 months ago

@hga007 Thank you for your detailed response. We will take a look at the test cases you mentioned to see what case we can port in UFS-Coastal. @uturuncoglu Ufuk, please chime in.

hga007 commented 10 months ago

Perhaps, new test cases need to be developed that are simpler. However, in my experience, you always have to come back and rethink everything because of the challenging aspects of real complex applications that involve land/sea masks and incongruent grids during coupling. I learned my lesson well when using ESMF_FieldRegridStore in coupled applications. I used Hurricane IRENE because it is a short simulation, and we have ocean observations.

John Warner has several test cases with COAWST using MCT, but they have more physics using waves, vortex force, sediment, and BBL.

I always wanted to set up an application for Lake Erie that can include ice modeling. I wonder if CICE6 can handle that application. It is undoubtedly more straightforward and doesn't have open boundary conditions. I build a grid for Lake Michigan, but I never use it. I think Lake Erie or Lake Ontario will be better because of the smaller grids. NOAA has great applications for the Great Lakes. What do you think? Of course, I will have to add it to my TODO list.

image
pvelissariou1 commented 10 months ago

@hga007 I think a Lake Erie test case with cice will be great Anyway CICE (3-4) is already in use with GLOFS and I think CICE6 should not be an issue either. We do have a lot of experience with modeling in lakes Erie and Michigan here so we can help as well. @uturuncoglu , @saeed-moghimi-noaa what do you think?

hga007 commented 10 months ago

For testing the coupling with CDEPS, we should use the Forward subdirectory test for the IRENE case. It uses the DOPPIO/MARACOOS grid (240x104x40) for a 42 hours hindcast. It is thoroughly tested! It runs fast on 12 processors. The Data/FRC subdirectory provides all the necessary input atmospheric fields to be ingested into CDEPS. It cannot get simpler than that. We even offer all the configuration files and the GNU and CMake build scripts.

The ROMS grid is shown here:

https://www.myroms.org/wiki/Model_Coupling_IRENE

By the way, the simulation length can be shortened to 6 or 12 hours.

uturuncoglu commented 10 months ago

@pvelissariou1 Are you plaining to put ROMS tests to CoastalApp-testsuite? If so, you need to define ROMS as a component under CoastalApp. Right? Maybe going in that road is not efficient since we need to do the similar things in both side (ufs-coastal and CoastalApp). So, maybe it is better to define those test directly under ufs-coastal. @hga007 do we have input files ready in somewhere? If so we could start to bring them and see what happens. But before that I need to integrate ROMS build system into ufs-coastal. I'll start working on that and see what happens.

uturuncoglu commented 10 months ago

@hga007 Okay. I found the files. https://github.com/myroms/roms_test/tree/main/IRENE

pvelissariou1 commented 10 months ago

@uturuncoglu We will work on ROMS only in UFS-Coastal at this point. Regarding the test cases let's have them in UFS-Coastal following I guess a similar build structure as with the rest test cases.

uturuncoglu commented 10 months ago

@hga007 Is there any documentation about cmake build. I think the custom applications needs to provide header file (with -DROMS_HEADER= option) that includes all the options but it seems that the existing test cases like IRENE can be build without it. Right? If so, which options do I need to pass to CMake command. If you give more information that would be great.

uturuncoglu commented 10 months ago

@pvelissariou1 Okay. That is great. Thanks

hga007 commented 10 months ago

We recommend having two macro variables to be passed to the CMake command or YAML configuration file. We use that approach in ROMS-JEDI. For example, we pass -DROMS_APP=IRENE and -DROMS_APP_DIR=/home/.../roms/IRENE. You also need to specify the location of the cloned ROMS source code in -DROMS_SRC_DIR. In this way, the ROMS source and ROMS application are independent of each other. Recall that ROMS is configured according to the application CPP.

Then, in the main UFS CMakeLists.txt file, the following block can be added

message( STATUS "---------------------------------------------------------" )
message( STATUS "ROMS Application:" )
message( STATUS "---------------------------------------------------------" )

if( NOT ROMS_APP AND NOT ROMS_APP_DIR )
  message( STATUS "Using ROMS default application" )

  set( ROMS_APP IRENE )
  set( ROMS_APP_DIR  /home/.../roms/IRENE )

  set( ROMS_UFS_APPLICATION     ${ROMS_APP} )
  set( ROMS_UFS_APPLICATION_DIR ${ROMS_APP_DIR} )

elseif( ROMS_APP AND NOT ROMS_APP_DIR )
  message( ERROR "If you set ROMS_APP you must also set ROMS_APP_DIR to
                  the absolute path containing your application configuration." )
else()
  set( ROMS_UFS_APPLICATION     ${ROMS_APP} )
  set( ROMS_UFS_APPLICATION_DIR ${ROMS_APP_DIR} )
endif()

message( STATUS "ROMS_UFS_APPLICATION=${ROMS_UFS_APPLICATION}" )
message( STATUS "ROMS_UFS_APPLICATION_DIR=${ROMS_UFS_APPLICATION_DIR}" )

set( MY_HEADER_DIR ${ROMS_APP_DIR} )
set( MY_CPP_FLAGS ANALYTICAL_DIR='${ROMS_APP_DIR}';HEADER_DIR='${ROMS_APP_DIR}';ROOT_DIR='${ROMS_SRC_DIR}';MPI )
set( LIBTYPE SHARED )
set( ROMS_EXECUTABLE OFF )
set( MPI ON )

Then, ROMS main CMakeList.txt will take care of the rest. The other issue is that ROMS needs to know what compiler is used. It can use the value of the FORT macro or whatever CMake detected during configuration from Stack-Spak.

I hope that this template is useful for adding ROMS to the UFS CMake configuration.

uturuncoglu commented 10 months ago

@hga007 Thanks Hernan. That is really useful. I think it is better to create very light weight CMake layer top of ROMS rather than putting these detail into top level CMake file. UFS generally keeping that file very lightweight and create the link between UFS and component itself through the extra layer like ROMS-Interface, which will belong to ufs-coastal level. Anyway, let me work on it by referring the information that you gave. I'll update you about it.

hga007 commented 10 months ago

I don't know if adding that lightweight layer to the ROMS source code is a good idea. It may break ROMS CMake capabilities that are already complicated. However, you can add that lightweight layer to the ROMS application directory. I understood that the UFS can be configured with a YAML file. That's a good place to setting those macros too.

We can have a similar configuration for Lake Erie to test the coupling to CICE6 or any other application that we want to connect to the UFS.

The directory Data/HyCOM in IRENE includes the SST temperature NetCDF used by the DATA component to meld ROMS and HyCOM exported SST needed by the atmospheric model. Notice that Data/WRF has the irene_wrf_meld_weights.nc file, which includes the melding coefficients for a smooth transition between ROMS and HyCOM imported SST by the atmosphere component. I don't know how CDEPS do such melding.

uturuncoglu commented 10 months ago

@hga It would be part of ufs-coastal not the ROMS. That layer will populate extra information required for ROMS Cmake and create the connection. Since ROMS has CMake built it would be very small layer. At this point, UFS can not be configured with a YAML file. This is the next step when we bring ESMX into the picture but we have still time for it. ESMX is more mature now but still might need extra development to use under such as complex application like UFS. I used it with CI to couple land and cdeps but that is all. Let me implement something for ROMs build and you could review it. In any case, we could change the approach or modify later once could build ROMS under UFS.

hga007 commented 10 months ago

Sounds good. I am updating the GitHub repositories today. It is the last thing I must do before making a public announcement.

I forgot to mention one issue in the IRENE configuration: I also specified a few CPP options in the cbuild_roms scripts that are not included in the frozen header file irene.h. It is done for research purposes to turn on/off specific options. I don't know how you want to do this in the lightweight layer. Check IRENE/Forward/cbuild_roms.sh definition of _MY_CPPFLAGS.

Notice that we are discussing ROMS native WorkFlow design. It is simple, but it is convenient and works well.

uturuncoglu commented 10 months ago

@hga007 Okay. Thanks for extra information. I'll update you about it. What do you mean about WorkFlow? Is it a tool that will configure input files, grid etc. for ROMS and run it?

hga007 commented 10 months ago

Nope, it's simpler than that. It's a strategy for how to run any ROMS application. The build script also has the -b option to clone specific feature branches into the project directory's local src subdirectory. All the compiled files got the Build_roms subdirectory.

uturuncoglu commented 10 months ago

@hga007 Do you have filing about the future size of the https://github.com/myroms/roms_test repository. At this point, it is very small (~38M) and I am planning to add it as a submodule along with ROMS since building existing tests case requires that files. Right? The other option could be checking out the test repository to the UFS input directory. In the second approach, the CMake layer (that will sit top of ROMS CMake build) might need to know about the UFS input directory which is hidden in the testing directory and I am not sure that will be a right solution. Anyway, please let me know what do you think?

uturuncoglu commented 10 months ago

@hga007 I could able to trigger build but it seems that it is using wrong MPI. In the following code,

https://github.com/myroms/roms_test/blob/bd8ed7f4f5789f0437f45dff39fccbd981b04eca/IRENE/Forward/cbuild_roms.sh#L245

It seems that MPI is fixed to which_MPI and the files in the ROMS/Compilers directory also looks to which_MPI. Is it possible to pass that information to ROMS build. It seems that even if I set an environment variable like which_MPI, it will be overwritten by those scripts. Maybe that can be done as a part of the ROMS/Compilers/roms_compiler_flags.cmake. Any idea?

uturuncoglu commented 10 months ago

@hga007 BTW, is ROMS CMake system builds the code in the source directory. Is it possible to point a custom directory for build?

hga007 commented 10 months ago

@uturuncoglu, Where do you get that number? The entire roms_test repository on GitHub is 2.6G, and IRENE is 833M. It is because several Test Cases include the initial configuration of NetCDF files. We pay annual fees to GitHub for the myroms Teams account and 50G of extra storage. It is not a free account. Rutgers declined for us to have a GitHub account because of cybersecurity issues. It took a lot of convincing to be approved. Thus, we have a Teams account because it is more secure. Let's hope Rutgers doesn't require an annual evaluation since we need to fill out forms.

Can you be more specific about the submodel? Do you plan to add that to the ROMS source code repository? I feel uneasy about that one.

I will foresee adding more realistic test cases, including NetCDF files. We certainly can add more stuff (subdirectories) to the roms_test repository if you need it. Alternatively, we can add a standalone repository to myroms, if that works better for you.

In ROMS-JEDI, a CMakeList.txt file seats in the bundle, which manages all the JEDI components, including ROMS. See https://github.com/JCSDA-internal/roms-jedi/blob/develop/bundle/CMakeLists.txt

Thus, that layer may reside in other repositories. It uses ecbuild (developed at ECMWF) to manage all the components. Since the UFS is moving to JEDI, they must use ecbuild. I use the following command to configure any ROMS-JEDI application:

ecbuild -DMPIEXEC_EXECUTABLE=$MPIRUN -DMPIEXEC_NUMPROC_FLAG="-n" -DMPIEXEC_NUMPROC=12 -DROMS_APP=DOPPIO -DROMS_APP_DIR=DOPPIOpath -DCMAKE_BUILD_TYPE=Release ../Bundle_doppio

Therefore, I recommend using a bundle that attaches and tests all the components to the UFS using ctest capabilities for regression. it is definitely the way to go. It may raise some bureaucratic constraints.

hga007 commented 10 months ago

It is not the case for us. It always builds in the project directory. Something is missing. Maybe the macro BUILD_DIR is empty. See cbuild_roms.sh for more information.

uturuncoglu commented 10 months ago

@hga007 I cloned https://github.com/myroms/roms_test.git and use du -hs under the directory and it just shows 38Mfor me. The clone also pretty fast. Do I need to use --recursive in git clone. If the size is such like that then it would make sense to have it under input directory. I think 'IRENE' is fine to test the build at this point.

hga007 commented 10 months ago

Hmm. We don't use which_MPI in JEDI. Are you using spack-stack?

hga007 commented 10 months ago

I bet that your configuration of GIT does not have LFS support! It is needed to download the NetCDF files correctly. Could you check the README.md for ROMS in GitHub? I put instructions for it.

uturuncoglu commented 10 months ago

@hga007 Okay. That explains. Okay, I'll check the README. I am trying to finalize the build. Once it is working, I'll point UFS input directory for ROMS. I still need to solve,

(1) passing correct MPI to ROMS (2) ability to build ROMS outside of its source directory

uturuncoglu commented 10 months ago

@hga007 It is not spack-stack. The UFS still using hpc-stack for the dependencies but it is not related with building the component. That information (like which_MPI) needs to be passed through the UFS top level CMake and then ROMS-Interface layer. At this point, I am testing build on Orion and it uses Intel for both compiler and MPI. The ROMS/Compilers/Linux-ifort.mk file uses mpif90 which is GNU on Orion. So, I need to pass which_MPI as intel to sue correct compiler (mpiifort).

pvelissariou1 commented 10 months ago

@hga007 , @uturuncoglu We don't LFS support for our GitHub repos. Usually we store the data on the cloud and the user is responsible to fetch the data before running model simulations. Ufuk is this a good option for UFS-Coastal RTs?

uturuncoglu commented 10 months ago

@hga007 I checked out with LFS and not I could see 1.6G for total size. Thanks for your help.

hga007 commented 10 months ago

@uturuncoglu Did you set MPI=ON ?

pvelissariou1 commented 10 months ago

@uturuncoglu In IMPI mpif90 is also available

uturuncoglu commented 10 months ago

@hga007 Yes. MPI is ON. @pvelissariou1 If I build as it is I am getting following error. As you can see, it uses gfortran not ifort. I think the key is to provide which_MPI to ROMS build. Working on it.

cd /work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS && mpif90 -DANALYTICAL_DIR=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Functionals\" -DANALYTICAL_DIR="'/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'" -DHEADER=\"irene.h\" -DHEADER_DIR="'/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'" -DIFORT -DIRENE -DLINUX -DMPI -DROMS_HEADER=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling/irene.h\" -DROOT_DIR=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS\" -DROOT_DIR="''" -DSVN_REV=\"\" -DSVN_URL=\"\" -DX86_64 -I/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS -I/usr/local/include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/Master -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Drivers -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Functionals -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Modules -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/BBL -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/Biology -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/Sediment -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Utility -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -nowarn -sox -align array64byte -qno-opt-dynamic-align -real-size 64 -fp-model precise -ip -O3 -traceback -check uninit -module modules -c /work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS/f90/array_modes.f90 -o CMakeFiles/Objects.dir/f90/array_modes.f90.o
gfortran: error: byterecl: No such file or directory
gfortran: error: array64byte: No such file or directory
gfortran: error: 64: No such file or directory
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-fpp’
gfortran: error: unrecognized command line option ‘-fno-alias’
gfortran: error: unrecognized command line option ‘-auto’
gfortran: error: unrecognized command line option ‘-safe-cray-ptr’
gfortran: error: unrecognized command line option ‘-ftz’
gfortran: error: unrecognized command line option ‘-assume’
gfortran: error: unrecognized command line option ‘-nowarn’
gfortran: error: unrecognized command line option ‘-sox’
gfortran: error: unrecognized command line option ‘-align’
gfortran: error: unrecognized command line option ‘-qno-opt-dynamic-align’
gfortran: error: unrecognized command line option ‘-real-size’
gfortran: error: unrecognized command line option ‘-fp-model’
gfortran: error: unrecognized command line option ‘-ip’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-check’
gfortran: error: unrecognized command line option ‘-module’
gfortran: error: byterecl: No such file or directory
gfortran: error: array64byte: No such file or directory
gfortran: error: 64: No such file or directory
gfortran: error: byterecl: No such file or directory
gfortran: error: precise: No such file or directory
gfortran: error: array64byte: No such file or directory
gfortran: error: uninit: No such file or directory
gfortran: error: 64: No such file or directory
gfortran: error: precise: No such file or directory
gfortran: error: uninit: No such file or directory
make[2]: *** [ROMS-interface/ROMS/CMakeFiles/Objects.dir/f90/ADfromTL.f90.o] Error 1
pvelissariou1 commented 10 months ago

@uturuncoglu Thanks

hga007 commented 10 months ago

Hi Ufuk,

Please send us the CMakeList.txt that you are using. You cannot mix CMake with cbuild_roms. The compiler and CMake are very confused.

Best, H

From: Ufuk Turunçoğlu @.> Date: Wednesday, August 16, 2023 at 3:21 PM To: oceanmodeling/ufs-coastal @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] ROMS Integration (Issue #9)

@hga007https://github.com/hga007 Yes. MPI is ON. @pvelissariou1https://github.com/pvelissariou1 If I build as it is I am getting following error. As you can see, it uses gfortran not ifort. I think the key is to provide which_MPI to ROMS build. Working on it.

cd /work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS && mpif90 -DANALYTICAL_DIR=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Functionals\" -DANALYTICAL_DIR="'/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'" -DHEADER=\"irene.h\" -DHEADER_DIR="'/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'" -DIFORT -DIRENE -DLINUX -DMPI -DROMS_HEADER=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling/irene.h\" -DROOT_DIR=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS\" -DROOT_DIR="''" -DSVN_REV=\"\" -DSVN_URL=\"\" -DX86_64 -I/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS -I/usr/local/include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/Master -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Drivers -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Functionals -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Modules -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/BBL -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/Biology -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/Sediment -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Utility -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -nowarn -sox -align array64byte -qno-opt-dynamic-align -real-size 64 -fp-model precise -ip -O3 -traceback -check uninit -module modules -c /work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS/f90/array_modes.f90 -o CMakeFiles/Objects.dir/f90/array_modes.f90.o

gfortran: error: byterecl: No such file or directory

gfortran: error: array64byte: No such file or directory

gfortran: error: 64: No such file or directory

gfortran: error: precise: No such file or directory

gfortran: error: uninit: No such file or directory

gfortran: error: unrecognized command line option ‘-traceback’

gfortran: error: unrecognized command line option ‘-fpp’

gfortran: error: unrecognized command line option ‘-fno-alias’

gfortran: error: unrecognized command line option ‘-auto’

gfortran: error: unrecognized command line option ‘-safe-cray-ptr’

gfortran: error: unrecognized command line option ‘-ftz’

gfortran: error: unrecognized command line option ‘-assume’

gfortran: error: unrecognized command line option ‘-nowarn’

gfortran: error: unrecognized command line option ‘-sox’

gfortran: error: unrecognized command line option ‘-align’

gfortran: error: unrecognized command line option ‘-qno-opt-dynamic-align’

gfortran: error: unrecognized command line option ‘-real-size’

gfortran: error: unrecognized command line option ‘-fp-model’

gfortran: error: unrecognized command line option ‘-ip’

gfortran: error: unrecognized command line option ‘-traceback’

gfortran: error: unrecognized command line option ‘-check’

gfortran: error: unrecognized command line option ‘-module’

gfortran: error: byterecl: No such file or directory

gfortran: error: array64byte: No such file or directory

gfortran: error: 64: No such file or directory

gfortran: error: byterecl: No such file or directory

gfortran: error: precise: No such file or directory

gfortran: error: array64byte: No such file or directory

gfortran: error: uninit: No such file or directory

gfortran: error: 64: No such file or directory

gfortran: error: precise: No such file or directory

gfortran: error: uninit: No such file or directory

make[2]: *** [ROMS-interface/ROMS/CMakeFiles/Objects.dir/f90/ADfromTL.f90.o] Error 1

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/9#issuecomment-1681150922, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TADMJU45WF2HIM736NTXVUMRVANCNFSM6AAAAAA3P47VH4. You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 10 months ago

@hga007 Here is my top level CMakeList.txt file, 

###############################################################################
### Set variables
###############################################################################

if(NOT ROMS_APP AND NOT ROMS_APP_DIR)
  message(STATUS "Using ROMS default application")

  set(ROMS_APP IRENE)
  set(ROMS_APP_DIR  ${CMAKE_CURRENT_SOURCE_DIR}/ROMS-test/IRENE/Coupling)

  set(ROMS_UFS_APPLICATION     ${ROMS_APP})
  set(ROMS_UFS_APPLICATION_DIR ${ROMS_APP_DIR})

elseif( ROMS_APP AND NOT ROMS_APP_DIR)
  message(ERROR "If you set ROMS_APP you must also set ROMS_APP_DIR to
                  the absolute path containing your application configuration." )
else()
  set(ROMS_UFS_APPLICATION     ${ROMS_APP})
  set(ROMS_UFS_APPLICATION_DIR ${ROMS_APP_DIR})
endif()

message(STATUS "ROMS_UFS_APPLICATION=${ROMS_UFS_APPLICATION}")
message(STATUS "ROMS_UFS_APPLICATION_DIR=${ROMS_UFS_APPLICATION_DIR}")

set(MY_HEADER_DIR ${ROMS_APP_DIR})
set(MY_CPP_FLAGS ANALYTICAL_DIR='${ROMS_APP_DIR}';HEADER_DIR='${ROMS_APP_DIR}';ROOT_DIR='${ROMS_SRC_DIR}';MPI)
set(LIBTYPE STATIC)
set(ROMS_EXECUTABLE OFF)
set(MPI ON)

if(NetCDF_Fortran_FOUND)
  set(ENV{NETCDF_INCDIR} "${NetCDF_Fortran_INCLUDE_DIRS}")
  set(ENV{NETCDF_LIBDIR} "${NetCDF_Fortran_LIBRARIES}")
endif()

# just for testing
#set(ENV{USE_MPIF90} "ON")
#set(ENV{which_MPI} "intel")

###############################################################################
### File lists and macros
###############################################################################

add_subdirectory(ROMS)
hga007 commented 10 months ago

How did you set ROMS_SRC_DIR ?

You need to make a project directory wherever you want and have the source code somewhere else.

Are you running cmake from CMAKE_CURRENT_SOURCE_DIR?

How cmake know what compiler is using?

You error mixes gfortran with ifort flags.

H

From: Ufuk Turunçoğlu @.> Date: Wednesday, August 16, 2023 at 3:40 PM To: oceanmodeling/ufs-coastal @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] ROMS Integration (Issue #9)

@hga007https://github.com/hga007 Here is my top level CMakeList.txt file,

###############################################################################

Set variables

###############################################################################

if(NOT ROMS_APP AND NOT ROMS_APP_DIR)

message(STATUS "Using ROMS default application")

set(ROMS_APP IRENE)

set(ROMS_APP_DIR ${CMAKE_CURRENT_SOURCE_DIR}/ROMS-test/IRENE/Coupling)

set(ROMS_UFS_APPLICATION ${ROMS_APP})

set(ROMS_UFS_APPLICATION_DIR ${ROMS_APP_DIR})

elseif( ROMS_APP AND NOT ROMS_APP_DIR)

message(ERROR "If you set ROMS_APP you must also set ROMS_APP_DIR to

              the absolute path containing your application configuration." )

else()

set(ROMS_UFS_APPLICATION ${ROMS_APP})

set(ROMS_UFS_APPLICATION_DIR ${ROMS_APP_DIR})

endif()

message(STATUS "ROMS_UFS_APPLICATION=${ROMS_UFS_APPLICATION}")

message(STATUS "ROMS_UFS_APPLICATION_DIR=${ROMS_UFS_APPLICATION_DIR}")

set(MY_HEADER_DIR ${ROMS_APP_DIR})

set(MY_CPP_FLAGS ANALYTICAL_DIR='${ROMS_APP_DIR}';HEADER_DIR='${ROMS_APP_DIR}';ROOT_DIR='${ROMS_SRC_DIR}';MPI)

set(LIBTYPE STATIC)

set(ROMS_EXECUTABLE OFF)

set(MPI ON)

if(NetCDF_Fortran_FOUND)

set(ENV{NETCDF_INCDIR} "${NetCDF_Fortran_INCLUDE_DIRS}")

set(ENV{NETCDF_LIBDIR} "${NetCDF_Fortran_LIBRARIES}")

endif()

just for testing

set(ENV{USE_MPIF90} "ON")

set(ENV{which_MPI} "intel")

###############################################################################

File lists and macros

###############################################################################

add_subdirectory(ROMS)

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/9#issuecomment-1681173694, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TAFKCZLBJQOOZMBY34LXVUO23ANCNFSM6AAAAAA3P47VH4. You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 10 months ago

@hga007 Here are my comments,

hga007 commented 10 months ago

Hi Ufuk,

Sorry, this can be solved very quickly by having a chat.

Yes, you need to specify ROMS_SRC_DIR.

Dave and I have been chatting and we don’t understand why are you trying to do. Want computer are you using and how the compiler are set up? Module load?

The ROMS CMake check the CMAKE_Fortran_COMPILER_ID. How that is set? Can you try:

cmake -DCMAKE_Fortran_COMPILER=gfortran -DROMS_SRC_DIR=???

H

From: Ufuk Turunçoğlu @.> Date: Wednesday, August 16, 2023 at 4:00 PM To: oceanmodeling/ufs-coastal @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] ROMS Integration (Issue #9)

@hga007https://github.com/hga007 Here are my comments,

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/9#issuecomment-1681196871, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TAEOXSAQZ2RZ56CMAMDXVURHDANCNFSM6AAAAAA3P47VH4. You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 10 months ago

@hga007 Okay. I think I solved it. I put definition of compiler to the beginning of the CMake file and also set COMM variable for Intel like following,

###############################################################################
### Fortran compiler flags
###############################################################################

if(CMAKE_Fortran_COMPILER_ID MATCHES "GNU")
  set(CMAKE_Fortran_FLAGS "-g -fbacktrace")
  set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -Waliasing -fcray-pointer -fconvert=big-endian -fno-range-check -fbacktrace")
  set(CMAKE_Fortran_FLAGS_RELEASE "-O2")
  set(CMAKE_Fortran_FLAGS_DEBUG "-O0 -fcheck=bounds -ffpe-trap=invalid,zero,overflow,underflow" )
  set(CMAKE_Fortran_LINK_FLAGS "")
elseif(CMAKE_Fortran_COMPILER_ID MATCHES "Intel")
  set(CMAKE_Fortran_FLAGS "-g -traceback")
  set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox")
  set(CMAKE_Fortran_FLAGS_RELEASE "-O2 -debug minimal -fp-model source")
  set(CMAKE_Fortran_FLAGS_DEBUG "-O0 -check -check noarg_temp_created -check nopointer -fpe0 -ftrapuv -init=snan,arrays")
  set(CMAKE_Fortran_LINK_FLAGS "")
  set(C_PREPROCESS_FLAG "-cpp")
  set(COMM "intel")
else()
  message(WARNING "Fortran compiler with ID ${CMAKE_Fortran_COMPILER_ID} will be used with CMake default options")
endif()

It is compiling now but I have error in the link step which I think it is caused because I am building under source directory. If I could force it to build custom place, I think that will solve the issue. I'll keep updating you.

hga007 commented 10 months ago

Hi Ufuk,

In the regular ROMS, the build script makes the subdirectory Build_romsM (macro BUILD_DIR). Then, cd ${BUILD_DIR} and execute make. The source code can be in Turkey, LOL, if you want using ROMS_SRC_DIR.

H

From: Ufuk Turunçoğlu @.> Date: Wednesday, August 16, 2023 at 4:14 PM To: oceanmodeling/ufs-coastal @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] ROMS Integration (Issue #9)

@hga007https://github.com/hga007 Okay. I think I solved it. I put definition of compiler to the beginning of the CMake file and also set COMM variable for Intel like following,

###############################################################################

Fortran compiler flags

###############################################################################

if(CMAKE_Fortran_COMPILER_ID MATCHES "GNU")

set(CMAKE_Fortran_FLAGS "-g -fbacktrace")

set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -Waliasing -fcray-pointer -fconvert=big-endian -fno-range-check -fbacktrace")

set(CMAKE_Fortran_FLAGS_RELEASE "-O2")

set(CMAKE_Fortran_FLAGS_DEBUG "-O0 -fcheck=bounds -ffpe-trap=invalid,zero,overflow,underflow" )

set(CMAKE_Fortran_LINK_FLAGS "")

elseif(CMAKE_Fortran_COMPILER_ID MATCHES "Intel")

set(CMAKE_Fortran_FLAGS "-g -traceback")

set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox")

set(CMAKE_Fortran_FLAGS_RELEASE "-O2 -debug minimal -fp-model source")

set(CMAKE_Fortran_FLAGS_DEBUG "-O0 -check -check noarg_temp_created -check nopointer -fpe0 -ftrapuv -init=snan,arrays")

set(CMAKE_Fortran_LINK_FLAGS "")

set(C_PREPROCESS_FLAG "-cpp")

set(COMM "intel")

else()

message(WARNING "Fortran compiler with ID ${CMAKE_Fortran_COMPILER_ID} will be used with CMake default options")

endif()

It is compiling now but I have error in the link step which I think it is caused because I am building under source directory. If I could force it to build custom place, I think that will solve the issue. I'll keep updating you.

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/9#issuecomment-1681212246, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TABF762DKJXXG2L65QTXVUS2TANCNFSM6AAAAAA3P47VH4. You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 10 months ago

@hga007 Okay. Thanks. I think I am very close to build app. The only missing thing is that UFS could not find the SetServices method. I check the created static library (libROMS.a) with nm command and it does not have SetServices method but I could see esmf_roms_mod.mod and cmeps_roms_mod.mod in the module directory (I am not sure which one needs to be used under UFS - probably cmeps one). I think I also need to set -DESMF_LIB. Right?

uturuncoglu commented 10 months ago

@hga007 Okay. I think I found it. Some of the variables are defined under https://github.com/myroms/roms_test/blob/main/IRENE/Coupling/build_roms.sh. So, those needs to be provided by the configuration.

hga007 commented 10 months ago

Hi Ufuk,

You are building ROMS with the static library instead of shared. The commands that we send you DOES NOT build ROMS executable. Then, you use the libROMS.so in your driving program. However, let us compile ROMS coupling with cmake to check if we are missing something.

H

From: Ufuk Turunçoğlu @.> Date: Wednesday, August 16, 2023 at 4:30 PM To: oceanmodeling/ufs-coastal @.> Cc: Hernan Arango @.>, Mention @.> Subject: Re: [oceanmodeling/ufs-coastal] ROMS Integration (Issue #9)

@hga007https://github.com/hga007 Okay. I think I found it. Some of the variables are defined under https://github.com/myroms/roms_test/blob/main/IRENE/Coupling/build_roms.sh. So, those needs to be provided by the configuration.

— Reply to this email directly, view it on GitHubhttps://github.com/oceanmodeling/ufs-coastal/issues/9#issuecomment-1681229984, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFP6TACOHBSBVJVGBIZCTYTXVUUWDANCNFSM6AAAAAA3P47VH4. You are receiving this because you were mentioned.Message ID: @.***>

uturuncoglu commented 10 months ago

@hga007 It think it is almost working. The only issue is that I am getting following error from ROMS/ROMS/Modules/mod_strings.F. The reason is that I think my_fflags variable is corrupted.

<command-line>:30:0: error: macro names must be identifiers
<command-line>:30:0: error: macro names must be identifiers

My build command for this file is somthing like following,

cd /work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS && /bin/cpp -P --traditional-cpp -w -DWIND_MINUS_CURRENT -DESMF_LIB -DDATA_COUPLING -DWRF_COUPLING -DWRF_TIMEAVG -DFRC_COUPLING -DROMS_STDOUT -DVERIFICATION -DCOLLECT_ALLREDUCE -DOUT_DOUBLE -DLINUX -DX86_64 -DIFORT -DROOT_DIR=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS\" -DIRENE -DHEADER=\"irene.h\" -DANALYTICAL_DIR=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Functionals\" -DROMS_HEADER=\"/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling/irene.h\" -DSVN_URL=\"\" -DSVN_REV=\"\" -DMPI "-DANALYTICAL_DIR='/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'" "-DHEADER_DIR='/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'" "-DROOT_DIR=''" -DMPI "-DMY_OS='Linux'" "-DMY_CPU='x86_64'" "-DMY_FORT='ifort'" "-DMY_FC='mpiifort'" "-DMY_FFLAGS='-g -traceback -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/esmf/8.3.0b09/mod" -D/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/esmf/8.3.0b09/include "-D/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/include -fp-model precise -ip -O3 -traceback -check uninit'" -I/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS -I/usr/local/include -I/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/Master -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Drivers -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Functionals -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Include -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Modules -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/BBL -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/Biology -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Nonlinear/Sediment -I/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Utility /work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS/ROMS/Modules/mod_strings.F > /work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS/f90/mod_strings.f90

It seems that following part is corrupted,

DMY_FFLAGS='-g -traceback -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/esmf/8.3.0b09/mod" -D/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/esmf/8.3.0b09/include "-D/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/netcdf/4.7.4/include -fp-model precise -ip -O3 -traceback -check uninit'"

and as a result I have following in the result file,

        character (len=80)  :: my_fc = 'mpiifort'
        character (len=1024) :: my_fflags = '-g -traceback -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox -g -traceback -fp-model precise   -fPIC -O2 -fPIC -assume realloc_lhs -m64 -mcmodel=small -pthread -threads  -qopenmp /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/esmf/8.3.0b09/mod
!
        character (len=256) :: Adir = '/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'
        character (len=256) :: Hdir = '/work/noaa/nems/tufuk/COASTAL/ufs-coastal/ROMS-interface/ROMS-test/IRENE/Coupling'
        character (len=256) :: Hfile = "irene.h"
        character (len=256) :: Rdir = ''
!
      END MODULE mod_strings

There is missing ' symbol at the end of the my_fflags variable. There are some codes in ROMS/Compilers/roms_compiler_flags.cmake but ROMS-interface/ROMS/CMakeLists.txt I don't know how build system uses them.

BTW, in the top level CMake file I have following. This enables to use ESMF module files to compile ESMF related source. 

if (ESMF_FOUND)
  set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} ${ESMF_F90COMPILEPATHS}")
endif()

Maybe this is wrong approach. If so, how can I pass ESMF module directory to ROMS build?

uturuncoglu commented 10 months ago

@hga007 I fixed the issue by removing extra definition in the CMAKE_Fortran_FLAGS (i am not sure why it was creating issue in the first place, there might be a minor bug in those scripts) and use include_directories(${ESMF_F90COMPILEPATHS}).

I think I also need to change the IRENE configuration little bit to fit my needs. Since I am not interested in to run the case with active WRF and internal data components. I removed following from the configuration,

  add_definitions(-DDATA_COUPLING)
  add_definitions(-DWRF_COUPLING)
  add_definitions(-DWRF_TIMEAVG)

and add -DMODEL_COUPLING. I think this will allow me to couple ROMS with CDEPS eventually using CMEPS. Please confirm that this is the right setup to achieve ROMS+DATM coupling. But I am getting error like following,

/work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS/f90/coupler.f90(65): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [MOD_ESMF_ESM]
      USE mod_esmf_esm          ! ESM coupling structures and variables
----------^
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/tests/build_fv3_coastal/ROMS-interface/ROMS/f90/coupler.f90(221): error #6683: A kind type parameter must be a compile-time constant.   [I4B]
      integer (i4b) :: srcMaskVal, dstMaskVal

It seems this is normal since the model does not build mod_esmf_esm. I both define -DESMF_LIB and -DMODEL_COUPLING in my Cmake file. So, that must be fine but I could not find any piece of information about building ROMS/Master/mod_esmf_esm.F. It is also not listed in ROMS/Master/CMakeLists.txt. Any idea? Maybe my configuration is not supported at this point. Not sure.

I also pushed all the development to the following fork. So you could check the implementation,

https://github.com/oceanmodeling/ufs-coastal/blob/feature/coastal_app/ROMS-interface/CMakeLists.txt