Stand-alone CMake Build

[jhamman]

We (@negin513, @billsacks, @mvertens, and I) are working on the LILAC project. We are at the stage where we're working on the build infrastructure connecting LILAC to CTSM. What we'd like to be able to do is use CMake to build CTSM as a stand alone library. Something close to this would be an ideal function set:

git clone https://github.com/ESCOMP/ctsm.git
cd ctsm/src
cmake .

Although CTSM includes a top level CMakeLists.txt, it does not seem like it is intended to support this type of workflow. For example, the example above yields the following error.

CMake Error at CMakeLists.txt:4 (include):
  include could not find load file:

    CIME_initial_setup
...

I imagine there are reasons this isn't currently possible so I'm looking for a) some explanation of those and b) thoughts from those familiar with the CTSM build system (@ekluzek, @billsacks) on how the desired functionality could be achieved.

Note, I've also raised a similar issue with CIME: https://github.com/ESMCI/cime/issues/3226.

[ekluzek]

Hmmm. The thing is that the CMakeLists.txt files are used for the unit testing build system. So to use this for all of CTSM we'd need a way to distinguish between the build of everything and the unit tests.

[jedwards4b]

IMO - you should use the existing build mechanism provided by cime. This will get you moving quickly - and then consider a cmake build as a later feature.

[billsacks]

@jhamman - It will probably be easier to discuss this by phone, google hangout, etc., but some initial thoughts:

The CMake build system currently in CTSM was designed solely for the sake of unit testing, and is intended to be run via the run_tests.py script, as documented in src/README.unit_testing. I have imagined that, at some point, we'd also support a build of the CTSM library via CMake, but I have never given much thought to what it would take to support the two side-by-side.

There are a number of aspects of the CMake build that are specific to unit testing. Two that come to mind (but there may be others) are:

• The unit test build only requires building a fairly small subset of the whole code base. Building everything needed for CTSM will slow down the unit test build. However, I don't mind changing that so that the unit tests build everything.

• There are a number of cases where we build a stub / fake version of a given module (ncdio_pio comes to mind) rather than the real module in a unit testing context. This is partly to break messy dependencies, and partly so we can have more control in the unit tests. So we'd need some logic in the CMake build so that the real module is included in the real build, but the stub version is included in the unit test build.

[jhamman]

IMO - you should use the existing build mechanism provided by cime. This will get you moving quickly - and then consider a cmake build as a later feature.

We are at this point now. @negin513 has a system that follows an unsavory and brittle pattern. This is sufficient for development and early testing but will need to be reworked before the project is complete.

a) create a CIME case that includes CTSM b) build the case c) locate the CTSM library d) build LILAC linking to the CTSM library

One of the goals of LILAC is to allow CTSM to be used in contexts beyond CESM. One of LILAC's target atmospheres is WRF which has its own complicated build system. The idea of maintaining both a CIME build and WRF build with manual linking steps just doesn't seem sustainable.

It sounds like this will be discussed at the next CTSM-software meeting.

[jedwards4b]

This should not be brittle or fragile - it is possible to use the build mechanism without creating a cime/ctsm case. Perhaps a review of the current build methodology would help.

[ekluzek]

Note, that the cime build system is used to build offline tools. For example the gen_domain executable. The steps to make that (see cime/tools/mapping/gen_domain_files/INSTALL are like this below. I'm not saying this is the right solution. But, it does remove your first three awkward steps from your outline above. Note, to get the below to work there's the addition of two small files, Filepath and Makefile that are specific for gen_domain. You should be able to set this up similarly for ctsm to build it and lilac as a library that can then be linked into your executable.

(1) $ cd src
(2) $ ../../../configure --macros-format Makefile --mpilib mpi-serial
Bash users:
(3) $ (. ./.env_mach_specific.sh ; gmake)
csh users:
(3) $ (source ./.env_mach_specific.csh ; gmake)

[billsacks]

Just to clarify: I talked with @jedwards4b to make sure I understood his suggestion, what it would buy us and what it wouldn't. (And thanks @jedwards4b for making your suggestion!)

Using the cime build system, via the configure script that @ekluzek points to, would provide a mechanism for loading modules and setting compiler flags for a cime-supported machine / compiler. We would still need to have a Makefile or CMake-based build that actually builds the library (reading in the generated Macros.make or Macros.cmake file).

My sense is that we ideally want to let the atmosphere model's build system specify any module environment, compilation flags, etc., and also that we don't want to tie ourselves to just cime-supported machines. So my initial thought is that the cime build system may not help us here. But we can discuss this further.

[billsacks]

It just occurred to me: If it's not already being done, it's important that the cmake build defines the cpp token NDEBUG in a non-debug case (and, I believe, DEBUG in a debug case... this should be checked against a CESM build to confirm). There are probably other cpp tokens that are needed as well in the build of the share code, but the NDEBUG one just occurred to me as one that's important in order for performance to be good (there are some relatively expensive checks that are done if NDEBUG isn't set).

[billsacks]

I talked with @jhamman about this a bit today. A brief summary (Joe, feel free to add anything important that I'm missing):

Joe tried to reuse parts of the existing CMake-based build system in CTSM, but ran into trouble because there are various dependencies / assumptions that are true in a unit test build, but weren't being met in this standalone build. It feels like any elegant solution is going to require a lot of work, so we talked about various ugly solutions that could get the job done for now.

It seems like the best solution would be to have a top-level CMakeLists.txt in CTSM/src that builds all of the cime dependencies of CTSM (share code, pio, etc.) and then builds the CTSM library itself. For now this would not make any add_subdirectory calls, so we wouldn't have to worry about interference from the existing CMakeLists.txt files, other than the current top-level file. For the top-level file, for now we could have a giant conditional enclosing the whole thing so that one block of code is executed if we're building for the sake of unit tests, and a different block for a full standalone build. (Yes, extremely ugly. We can gradually iterate to merge these builds into one.) We wouldn't mess with cime for now; instead, all of the CMake code to build cime's stuff would reside in CTSM's CMakeLists.txt. (Again, ugly. Eventually we could try to push this down into cime.)

Eventually we might want to migrate to a more standard CMake style build, where you make add_subdirectory calls. To do this, we would need to unify the unit test & standalone builds (or have giant conditional blocks in the lower-level files), and we would also need to resolve the dependency issue (currently, there isn't a clean dependency structure between different subdirectories; we would need to reorganize directories to give this clean dependency structure if we want to use a more standard CMake-style build).

Given that this would be so non-CMake-like, we also talked about just doing this with a big Makefile, but felt that it would probably be at least marginally easier to do it with CMake than with Make.

[billsacks]

See also #855

[billsacks]

My current thinking is to do this with a wrapper to the cime build rather than creating a parallel build system. While I believe that a CMake-based build would be a reasonable option for just building CTSM once all of the cime pieces were already built, I'm increasingly feeling that it will be problematic to try to create and maintain a separate CMake-based build of all the cime pieces, many of which have a number of build-time options that we would need to maintain. See #855 and ESMCI/cime#3489 for more on this. I acknowledge that this fundamentally builds on what @jhamman characterized as "unsavory and brittle", though I feel that this can be done in a way that is much easier to use than what was in place last September. (I also acknowledge that it may be hard to debug problems when things go wrong for users with my proposal. But the same could be true to some extent of a CMake-based build.)

I acknowledge that this solution is far from perfect. But achieving a substantially more ideal solution – which would likely involve fundamentally reworking cime's build system as @jhamman suggested in https://github.com/ESMCI/cime/issues/3226 – would likely take order months rather than the couple of weeks that I can devote to this.

(I haven't completely settled on this new idea, though... I want to spend some time playing with it and hearing feedback before making a final decision.)

[billsacks]

Closing this in favor of the cime-based build system discussed in #855