search

LLNL / axom

CS infrastructure components for HPC applications
BSD 3-Clause "New" or "Revised" License
157 stars 27 forks source link

help avoiding unnecessary recompilation #1203

Closed samuelpmishLLNL closed 11 months ago

samuelpmishLLNL commented 1 year ago

In serac, whenever I touch any of our cmake files, axom (0da8a5b1b, built alongside serac with add_subdirectory) also gets reconfigured since it's one of our dependencies. Even if the change to the cmake file is unrelated to axom, it still triggers a complete rebuild of all of our axom components:

% make some_test
...
[build] [... 100%] Linking CXX executable tests/some_test

% touch /path/to/serac/some/directory/CMakeLists.txt && make some_test
...
[build] [14/109   0% :: 0.160] Building CXX object axom/axom/core/CMakeFiles/core.dir/Types.cpp.o
[build] [15/109   1% :: 0.638] Building CXX object axom/axom/core/CMakeFiles/core.dir/utilities/nvtx/interface.cpp.o
[build] [16/109   2% :: 0.710] Building CXX object axom/axom/core/CMakeFiles/core.dir/utilities/nvtx/Range.cpp.o
[build] [17/109   3% :: 0.792] Building CXX object axom/axom/core/CMakeFiles/core.dir/utilities/Utilities.cpp.o
[build] [18/109   4% :: 0.805] Building CXX object axom/axom/core/CMakeFiles/core.dir/numerics/polynomial_solvers.cpp.o
[build] [19/109   5% :: 1.009] Building CXX object axom/axom/slic/CMakeFiles/slic.dir/streams/SynchronizedStream.cpp.o
[build] [20/109   6% :: 1.049] Building CXX object axom/axom/slic/CMakeFiles/slic.dir/core/LogStream.cpp.o
[build] [21/109   7% :: 1.050] Building CXX object axom/axom/lumberjack/CMakeFiles/lumberjack.dir/BinaryTreeCommunicator.cpp.o
[build] [22/109   8% :: 1.084] Building CXX object axom/axom/core/CMakeFiles/core.dir/utilities/About.cpp.o
[build] [23/109   9% :: 1.169] Building CXX object axom/axom/slic/CMakeFiles/slic.dir/streams/LumberjackStream.cpp.o
[build] [24/109  10% :: 1.265] Building CXX object axom/axom/slic/CMakeFiles/slic.dir/streams/GenericOutputStream.cpp.o
[build] [25/109  11% :: 1.291] Building CXX object axom/axom/slic/CMakeFiles/slic.dir/interface/slic.cpp.o
[build] [26/109  11% :: 1.504] Building CXX object axom/axom/slic/CMakeFiles/slic.dir/core/Logger.cpp.o
[build] [27/109  12% :: 1.869] Building CXX object axom/axom/core/CMakeFiles/core.dir/utilities/StringUtilities.cpp.o
[build] [28/109  13% :: 2.179] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/Attribute.cpp.o
[build] [29/109  14% :: 2.195] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/spio/IOBaton.cpp.o
[build] [30/109  15% :: 2.270] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/AttrValues.cpp.o
[build] [31/109  16% :: 2.348] Building CXX object axom/axom/core/CMakeFiles/core.dir/Path.cpp.o
[build] [32/109  17% :: 2.428] Linking CXX static library lib/libaxom_core.a
[build] [33/109  18% :: 2.500] Linking CXX static library lib/libaxom_lumberjack.a
[build] [34/109  19% :: 2.580] Linking CXX static library lib/libaxom_slic.a
[build] [35/109  20% :: 2.813] Building CXX object axom/axom/slam/CMakeFiles/slam.dir/BitSet.cpp.o
[build] [36/109  21% :: 3.077] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/Buffer.cpp.o
[build] [37/109  22% :: 3.395] Building CXX object axom/axom/slam/CMakeFiles/slam.dir/OrderedSet.cpp.o
[build] [38/109  22% :: 3.467] Linking CXX static library lib/libaxom_slam.a
[build] [39/109  23% :: 3.521] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/View.cpp.o
[build] [40/109  24% :: 3.701] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/DataStore.cpp.o
[build] [41/109  25% :: 4.052] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/blueprint.cpp.o
[build] [42/109  26% :: 4.162] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/Group.cpp.o
[build] [43/109  27% :: 4.874] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/spio/IOManager.cpp.o
[build] [44/109  28% :: 5.208] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/fem/FiniteElement.cpp.o
[build] [45/109  29% :: 5.605] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/internal/MeshHelpers.cpp.o
[build] [46/109  30% :: 5.662] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/CurvilinearMesh.cpp.o
[build] [47/109  31% :: 6.071] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/FieldData.cpp.o
[build] [48/109  32% :: 6.074] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/MeshCoordinates.cpp.o
[build] [49/109  33% :: 6.337] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/ParticleMesh.cpp.o
[build] [50/109  33% :: 6.667] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/VariantKey.cpp.o
[build] [51/109  34% :: 6.843] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/StructuredMesh.cpp.o
[build] [52/109  35% :: 6.903] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/RectilinearMesh.cpp.o
[build] [53/109  36% :: 7.078] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/UniformMesh.cpp.o
[build] [54/109  37% :: 7.127] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/mesh/Mesh.cpp.o
[build] [55/109  38% :: 8.110] Building CXX object axom/axom/sidre/CMakeFiles/sidre.dir/core/MFEMSidreDataCollection.cpp.o
[build] [56/109  39% :: 8.244] Linking CXX static library lib/libaxom_sidre.a
[build] [57/109  40% :: 8.328] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/utils/su2_utils.cpp.o
[build] [58/109  41% :: 9.169] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/Inlet.cpp.o
[build] [59/109  42% :: 9.173] Building CXX object axom/axom/mint/CMakeFiles/mint.dir/utils/vtk_utils.cpp.o
[build] [60/109  43% :: 9.325] Linking CXX static library lib/libaxom_mint.a
[build] [61/109  44% :: 9.585] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/Proxy.cpp.o
[build] [62/109  44% :: 9.679] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/Function.cpp.o
[build] [63/109  45% :: 9.686] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/VerifiableScalar.cpp.o
[build] [64/109  46% :: 9.851] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/Geometry.cpp.o
[build] [65/109  47% :: 9.884] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/Field.cpp.o
[build] [66/109  48% :: 10.108] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/Shape.cpp.o
[build] [67/109  49% :: 10.209] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/GeometryOperators.cpp.o
[build] [68/109  50% :: 10.250] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/ShapeSet.cpp.o
[build] [69/109  51% :: 11.320] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/inlet_utils.cpp.o
[build] [70/109  52% :: 11.469] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/JSONSchemaWriter.cpp.o
[build] [71/109  53% :: 11.576] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/SphinxWriter.cpp.o
[build] [72/109  54% :: 11.963] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/ConduitReader.cpp.o
[build] [73/109  55% :: 12.025] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/Container.cpp.o
[build] [74/109  55% :: 12.221] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/AllNearestNeighbors.cpp.o
[build] [75/109  56% :: 12.640] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/KleeError.cpp.o
[build] [76/109  57% :: 13.385] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/Units.cpp.o
[build] [77/109  58% :: 13.727] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/SignedDistance.cpp.o
[build] [78/109  59% :: 14.214] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/IOUtil.cpp.o
[build] [79/109  60% :: 14.228] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/Discretize.cpp.o
[build] [80/109  61% :: 14.650] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/IO.cpp.o
[build] [81/109  62% :: 14.912] Building CXX object axom/axom/klee/CMakeFiles/klee.dir/GeometryOperatorsIO.cpp.o
[build] [82/109  63% :: 15.125] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/readers/STLReader.cpp.o
[build] [83/109  64% :: 15.243] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/readers/ProEReader.cpp.o
[build] [84/109  65% :: 15.614] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/util/mesh_helpers.cpp.o
[build] [85/109  66% :: 15.690] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/interface/internal/QuestHelpers.cpp.o
[build] [86/109  66% :: 16.240] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/MeshTester.cpp.o
[build] [87/109  67% :: 16.455] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/interface/signed_distance.cpp.o
[build] [88/109  68% :: 17.560] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/readers/PSTLReader.cpp.o
[build] [90/109  70% :: 18.075] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/readers/PProEReader.cpp.o
[build] [92/109  72% :: 18.240] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/detail/shaping/shaping_helpers.cpp.o
[build] [93/109  73% :: 18.340] Building CXX object axom/axom/multimat/CMakeFiles/multimat.dir/multimat.cpp.o
[build] [94/109  74% :: 18.426] Linking CXX static library lib/libaxom_multimat.a
[build] [95/109  75% :: 18.471] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/Shaper.cpp.o
[build] [96/109  76% :: 18.587] Building CXX object src/serac/infrastructure/CMakeFiles/serac_infrastructure.dir/initialize.cpp.o
[build] [97/109  77% :: 19.131] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/MarchingCubes.cpp.o
[build] [98/109  77% :: 19.176] Building CXX object src/serac/infrastructure/CMakeFiles/serac_infrastructure.dir/about.cpp.o
[build] [99/109  78% :: 19.184] Building CXX object src/serac/infrastructure/CMakeFiles/serac_infrastructure.dir/profiling.cpp.o
[build] [100/109  79% :: 19.568] Building CXX object axom/axom/inlet/CMakeFiles/inlet.dir/LuaReader.cpp.o
[build] [101/109  80% :: 19.770] Linking CXX static library lib/libaxom_inlet.a
[build] [102/109  81% :: 19.902] Linking CXX static library lib/libaxom_klee.a/terminator.cpp.o
[build] [103/109  83% :: 20.496] Building CXX object axom/axom/quest/CMakeFiles/quest.dir/interface/inout.cpp.o
[build] [104/109  84% :: 20.713] Linking CXX static library lib/libaxom_quest.a

Other targets built as part of this CMake project are unaffected (mfem, serac subcomponents, etc)-- it's only the axom targets that recompile.

I'm looking for a way to avoid paying this cost each time I interact with our build system. Additionally, when setting some of the optional components to OFF (we don't even use slam, quest, primal, klee, mint), linking against the axom target seems to still bring them them in on the link line, leading to compilation errors. The linkage error may be specific to serac, that part is still unclear to me.

Is it possible to prevent axom from recompiling everything when making a change to a CMake file?

samuelpmishLLNL commented 1 year ago

Here is one potential CMake project implementation for a library (e.g. axom) that optionally builds some combination of components based on user-specified options: library_of_components.zip

Making changes to the CMake project and reconfiguring doesn't result in a complete rebuild. If a user previously built this library and then decides to turn on one of the optional components, only that new component library is compiled (object files from the other components are reused, rather than recompiled). Making other changes to the CMake project that don't affect the component builds requires no recompilation.

white238 commented 11 months ago

It is my understanding that there are two parts to this issue and I will address them separately.

1) Turning off individual components

Axom has CMake options to allow turning off individual components. The options are documented here:

https://axom.readthedocs.io/en/develop/docs/sphinx/quickstart_guide/config_build.html#cmake-configuration-options

We cannot turn off components in Serac due to what Serac requires but more to the point Tribol requires all of Axom. I did create two PRs that limit the inheritance of the targets so, while they are all built, it does limit the need to link them up the chain.

2) Touching CMake files causes rebuilding of part of Axom

This is more tricky. I was able to confirm that in Serac's Codevelop build that part (not all) of Axom is rebuilt when you only touch a CMake file. This does not happen in straight Axom though.

I have looked over all the things we do to make the Codevelop build work and nothing stands out in touching or reconfiguring files which is usually the case.

If you can figure out why Serac's codevelop build triggers a partial rebuild, I would be glad for a fix.

But for now this should be closed because its not an Axom issue.

samuelpmishLLNL commented 11 months ago

I have looked over all the things we do to make the Codevelop build work and nothing stands out in touching or reconfiguring files which is usually the case.

If you can figure out why Serac's codevelop build triggers a partial rebuild, I would be glad for a fix.

ninja will tell you why it's recompiling individual files:

serac/build % ninja -d explain
...
ninja explain: axom/axom/mint/CMakeFiles/mint.dir/fem/FiniteElement.cpp.o is dirty
ninja explain: output axom/axom/mint/CMakeFiles/mint.dir/mesh/internal/MeshHelpers.cpp.o older than most recent input /home/sam/code/serac/build/mfem/mfem.hpp (1699386084663511236 vs 1699386110103460827)ninja explain: axom/axom/mint/CMakeFiles/mint.dir/mesh/internal/MeshHelpers.cpp.o is dirty
ninja explain: output axom/axom/mint/CMakeFiles/mint.dir/mesh/CurvilinearMesh.cpp.o older than most recent input /home/sam/code/serac/build/mfem/mfem.hpp (1699386084475511608 vs 1699386110103460827)
...

It seems mfem is just regenerating mfem.hpp every time and many of axom's sourcefiles include it, so they end up getting recompiled as a result. A simple fix (in mfem) would be to only emplace the generated mfem.hpp file into the build directory if its contents are different than what's there already.