Support exported CMake targets

[nilsvu]

This is a proposal to support CMake targets for linking Charm++, as opposed to using the charmc compiler wrapper.

In our numerical relativity code SpECTRE (https://github.com/sxs-collaboration/spectre) that is based on Charm++ we are currently attempting to link Charm++ as a CMake target, in order to make our build system more robust. This is similar to how MPI is linked in modern CMake build systems:

find_package(MPI REQUIRED)
target_link_libraries(MyExecutable PRIVATE MPI::MPI_CXX)

So instead of using the MPI compiler wrappers, the CMake target MPI::MPI_CXX defines all the include directories, library paths, compiler flags etc. and can be linked into executables that need it. The MPI::MPI_CXX target is defined by CMake's native FindMPI module as an "imported target". For SpECTRE we are currently writing a FindCharm module that defines a Charmxx::charmxx imported target, using the extremely helpful charmc -print-building-blocks feature (see https://github.com/sxs-collaboration/spectre/pull/2680). Then, we link Charm++ like this:

find_package(Charm 6.10.2 EXACT REQUIRED COMPONENTS EveryLB ScotchLB)
target_link_library(MyExecutable PRIVATE Charmxx::charmxx)

This is working quite well, but it would be even more robust if Charm++ would define the target and export it, using the build-system information it has natively. CMake explains imported and exported targets in their documentation:

• CMake Importing and Exporting Guide

I believe a CMake integration based on exported targets would make it significantly easier to build executables on top of Charm++. Since Charm++ has been moving to a CMake-based build system recently, the foundation for this feature is already there. If you agree with my proposal, I would be happy to help if I can.

[matthiasdiener]

Hi @nilsleiffischer, this is a great idea! If you would like, please feel free to open a PR to Charm++ with the FindCharm module and we'd be happy to add it.

[matthiasdiener]

Does #3547 / #3548 go in the right direction?

[NK-Nikunj]

Hi @nilsleiffischer, For us to not rely on charmc, additional functionalities are required (something akin to add_charm_executable as opposed to add_executable). But the recent additions will allow users to find charm++ using find_package(Charm) within CMake and set the CMAKE_CXX_COMPILER to charmc. It works well and we're using it to build charmlite. To use Charm++, the user requires to append CMAKE_PREFIX_PATH with the installation (or build) directory of Charm++. The users will also have access to the flags set by Charm++ in its build system to better assist with the flags required for the testing/benchmarking framework. I believe it is a step in the right direction. It will require tinkering with the whole build system but if there's enough support for it - I can implement linking to charm++ as opposed to utilizing charmc in general.

[nilsvu]

@matthiasdiener @NK-Nikunj This is excellent, thank you so much for your efforts in this direction! Here are a few things I noticed:

• I recommend you use CMake's configure_package_config_file and write_basic_package_version_file. See:
  • https://cmake.org/cmake/help/git-stage/guide/importing-exporting/index.html#creating-packages
  • https://cmake.org/cmake/help/git-stage/module/CMakePackageConfigHelpers.html
• I recommend you use exported CMake targets for everything like include directories, linked libs, etc. The keywords PRIVATE, PUBLIC and INTERFACE help control which properties propagate to the exported targets. See:
  • https://cmake.org/cmake/help/git-stage/guide/importing-exporting/index.html#exporting-targets
• Dependencies like MPI, Scotch etc can be found in the CMake config file and linked to the exported targets.
• In the FindCharm module that I'm writing (https://github.com/sxs-collaboration/spectre/pull/2680) I define the targets Charm::charm (all libs), Charm::pup (only PUP serialization) and Charm::main (for linking executables with a main function). Linking with Charm::charm is enough for things like our Python bindings and our parallelization libraries that use Charm++. More basic libraries that define data structures etc only need to link with Charm::pup. Executables link with Charm::main. Then, to eliminate charmc as the compiler wrapper only a few extra things are necessary for executables:
  • I generate moduleinit files by linking executables to a custom target CharmModuleInit, which invokes charmc with some dummy flags.
  • We have an add_charm_module CMake function that generates ${MODULE}.decl.h and ${MODULE}.def.h files from .ci files by invoking charmc.

All of this is just to give you an idea how we are currently handling things. Feel free to pick and choose what works best for you!

[NK-Nikunj]

I had a very similar implementation detail in mind wrt targets and exports. BTW, we also added the --install-prefix to install charm++ instead of building in the source. We're adding features to the build system in Charm++ as we progress with Charmlite and require cmake flags/features from Charm++.

The methods you suggested require a fair bit of code reinventions given the tight coupling in our build scripts but it's certainly achievable.

[nilsvu]

Sounds great @NK-Nikunj! I intend to test our build system with our FindCharm module for a while, and then see if we can contribute some of the features upstream to Charm++, coordinating with you of course.

[nilsvu]

@NK-Nikunj I looked at the charmlite repository a bit. Do I understand correctly that it is a more C++-native interface to Charm++, that is based on templates rather than .ci files, and that eliminating charmc in favor of the standard cmake compiler aligns well with the project? I think it looks super useful already, and personally I would love to see that sort of thing migrate to upstream Charm++ at some point, if this is something you are considering at all :)

[NK-Nikunj]

@nilsleiffischer Yes, charmlite is experimental work on top of converse that relies on template metaprogramming instead of .ci files. We do use charmc as the compiler (see this) but it can easily be switched to use default cmake compilers and targets once we have updated Charm++'s build system. Unfortunately, charmlite doesn't support most features yet and is only for research purposes to highlight the overheads within charm++. @jszaday should be able to elucidate more on what goes back into charm++.

[mabruzzo]

Out of curiosity, has there been any more recent consideration of this topic? It would be great to have a slightly more robust process for compiling/building charm++ applications in cmake with standard c++ compilers and linkers.

In Enzo-E, we currently overwrite CMAKE_CXX_LINK_EXECUTABLE to make use of charmc after we perform find_package(Charm)[^1].

• There's recently been a lot of discussion about linkers in the cmake community (due to recent changes in Apple's linker). While cmake does support custom linker-wrappers, I get the sense that modifying CMAKE_CXX_LINK_EXECUTABLE outside of a toolchain file or a CMakePresets.json file is frowned upon by the CMake developers (and it may not be officially supported). Consequently, there doesn't appear to do this "correctly" while also leveraging find_package in our build.
• With that said, I haven't seen anything that expressly forbids this approach (and the approach definitely works right now).

Adopting an approach like the one @nilsvu implemented in SpECTRE seems like a more robust solution. But, I think their solution manually re-implements some unexposed functionality from within charmc (e.g. #3210). Consequently, I'm a little hesitant to directly adopt their solution since it seems like future charm++ details could freely change this functionality. I would feel a lot more comfortable if charm++ directly provided this functionality (or at least was willing to test/support this approach).

Obviously, I totally understand if this isn't a priority at this time

[^1]: We have been using a custom FindCharm.cmake module, but I plan to shift to using the package-config file in the near future.