Use OASIS3-MCT directly from CERFACS (no Cmake!)

[joakimkjellsson]

Good evening all

It's been bugging me a while that in ESM-Tools we compile OpenIFS, NEMO etc as the developers intend the model to be compiled, but with OASIS we use some Cmake method that does not come from CERFACS.

It's my understanding that this CMake method relies on first downloading OASIS3-MCT, then running the ./configure script, and then introducing Cmakefiles in a few directories. This makes it less than straightforward to upgrade OASIS to a newer version, it relies on Cmake being available, and it was also hard coded to always use mpiifort and mpicc as compilers.

I've made a new version of OASIS in ESM-Tools (5.0_geomar) which uses a git repo at GEOMAR which is a direct clone of the CERFACS branch. This compiles using make as intended by the developers. This way, we can use OASIS exactly as it comes from CERFACS by only adding a make.ESMTOOLS script, similarly to the esm.fcm file in OpenIFS or the arch files for XIOS. I've written the comp_command in oasis3mct.yaml so that it makes a build directory and put the libraries exactly as they were from Cmake, thus it should be compatible with older versions.

This does not really change any of the science we do, but it's cleaner and allows us to stay better in sync with CERFACS. I've tested it with FOCI-OpenIFS on HLRN Göttingen and it compiles and runs fine. In case other modelling groups (AWI-CM users etc) would like to try this out, see branch feature/oasis_from_cerfacs. If people are happy with it, I would suggest this to be the standard for future model versions.

Cheers Joakim

[seb-wahl]

Great idea as adjusting for cmake can be a pain. I have done a similar thing for the "old" FOCI for oasis as a test case (for oasis2.8 this meant going back to ./configure and make. Minor suggestion for your make.ESMTOOLS: Also activate -g -traceback when debugging is off. It helps as you know when the model crashes.

[mandresm]

I like the idea, thanks for your contribution and the initiative @joakimkjellsson! We will discuss this in the next ESM-Tools meeting where @seb-wahl is around. I'll keep the issue open until we talk about it.

[JanStreffing]

This gave me a bit of a chuckle :D. Years ago I heard this conversation at AWI: "It's been bugging me a while that in AWI-CM we compile ECHAM, FESOM, PSIM etc all with CMAKE, but with OASIS we use some old make method that does not work the same way."

I agree upgrading oasis versions is not ideal. At least as long as we don't have a proper fork from the original repo with the fully history. I've done the upgrade a few times by now, so it does not seem that hard to me. But that's probably not a good argument.

If make works, I'd be ok with using it.

[joakimkjellsson]

Glad to hear we are on the same page.

For @seb-wahl, I've added -g -traceback to the compile flags.

The last missing part is to make sure esm_master clean-focioifs-2.2/oasis works. However, I'm struggling at the moment. To compile, i.e. esm_master comp-focioifs-2.2/oasis, ESM-Tools does this

    Executing commands in this order:
         mkdir -p focioifs-agrif-3.0
         cd focioifs-agrif-3.0
         sed -i '/COUPLENEMOECE = /s/.TRUE./.FALSE./g' oifs-43r3-foci22/src/ifs/module/yommcc.F90
         sed -i '/COUPLEFESOM2 = /s/.TRUE./.FALSE./g' oifs-43r3-foci22/src/ifs/module/yommcc.F90
         sed -i '/COUPLENEMOFOCI = /s/.FALSE./.TRUE./g' oifs-43r3-foci22/src/ifs/module/yommcc.F90
         cd oasis
         export ESM_OASIS_DIR=/home/shkjocke/esm/models/focioifs-agrif-3.0/oasis ; cd util/make_dir/ ; cp make.ESMTOOLS make.inc ; make -f TopMakefileOasis3 ; cd ../../ ; mv INSTALL_OASIS.ESMTOOLS/lib/libpsmile.MPI1.a INSTALL_OASIS.ESMTOOLS/lib/libpsmile.a ; mkdir -p build/lib/psmile/scrip ; mkdir -p build/lib/psmile/mct/mpeu ; cp INSTALL_OASIS.ESMTOOLS/build-static/lib/psmile.MPI1/*.mod build/lib/psmile/. ; cp INSTALL_OASIS.ESMTOOLS/build-static/lib/scrip/*mod build/lib/psmile/scrip/. ; cp INSTALL_OASIS.ESMTOOLS/build-static/lib/mct/*mod build/lib/psmile/mct/. ; cp INSTALL_OASIS.ESMTOOLS/lib/libmct.a build/lib/psmile/mct/. ; cp INSTALL_OASIS.ESMTOOLS/lib/libmpeu.a build/lib/psmile/mct/. ; cp INSTALL_OASIS.ESMTOOLS/lib/libpsmile.a build/lib/psmile/. ; cp INSTALL_OASIS.ESMTOOLS/lib/libscrip.a build/lib/psmile/scrip/.
         cd ..
         mkdir -p ./lib
         cp  oasis/INSTALL_OASIS.ESMTOOLS/lib/libpsmile.a  lib
         cp  oasis/INSTALL_OASIS.ESMTOOLS/lib/libmct.a  lib
         cp  oasis/INSTALL_OASIS.ESMTOOLS/lib/libmpeu.a  lib
         cp  oasis/INSTALL_OASIS.ESMTOOLS/lib/libscrip.a  lib
         cd ..

and it works.

To clean, I would like to do this

    Executing commands in this order:
         mkdir -p focioifs-agrif-3.0
         cd focioifs-agrif-3.0
         sed -i '/COUPLENEMOECE = /s/.TRUE./.FALSE./g' oifs-43r3-foci22/src/ifs/module/yommcc.F90
         sed -i '/COUPLEFESOM2 = /s/.TRUE./.FALSE./g' oifs-43r3-foci22/src/ifs/module/yommcc.F90
         sed -i '/COUPLENEMOFOCI = /s/.FALSE./.TRUE./g' oifs-43r3-foci22/src/ifs/module/yommcc.F90
         cd oasis
         export ESM_OASIS_DIR=/home/shkjocke/esm/models/focioifs-agrif-3.0/oasis ; cd util/make_dir/ ; make -f TopMakefileOasis3 realclean ; cd ../../ ; rm -rf build lib/libpsmile.a lib/libscrip.a lib/libmct.a lib/libmpeu.a
         cd ..
         rm -f ./lib/libpsmile.a
         rm -f ./lib/libmct.a
         rm -f ./lib/libmpeu.a
         rm -f ./lib/libscrip.a
         cd ..

but that fails with

Traceback (most recent call last):
  File "/home/shkjocke/miniconda3/bin/esm_master", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_master')())
  File "/home/shkjocke/esm/esm_tools/src/esm_master/cli.py", line 103, in main
    main_flow(parsed_args, target)
  File "/home/shkjocke/esm/esm_tools/src/esm_master/esm_master.py", line 107, in main_flow
    user_task.execute(ignore_errors)  # env)
  File "/home/shkjocke/esm/esm_tools/src/esm_master/task.py", line 514, in execute
    subprocess.run(
  File "/home/shkjocke/miniconda3/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/shkjocke/miniconda3/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/shkjocke/miniconda3/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'export'

which makes no sense. Somehow, it can run export command for comp, but not clean.

Do you have any ideas @mandresm on how to overcome this?

Cheers Joakim