search

sxs-collaboration / spectre

SpECTRE is a code for multi-scale, multi-physics problems in astrophysics and gravitational physics.
https://spectre-code.org
Other
160 stars 189 forks source link

Reduce memory usage of compiling executables #5472

Open nilsvu opened 1 year ago

nilsvu commented 1 year ago

Compiling executables needs a lot of RAM. Currently building the BBH executable with gcc 11 in Release mode on CI exceeds the 7GB of RAM available on the Github-hosted runners and fails. Also building in parallel often exceeds the available RAM on clusters and personal machines. Therefore, it would be very helpful to reduce the memory usage of our builds.

nilsvu commented 1 year ago

Here's a recent profile I did with clang's build analyzer: profile.txt

kidder commented 1 year ago

Here are the top offenders in compile-time and memory usage for a release build using gcc 13.1 on my desktop (with DEBUG_SYMBOLS=OFF, BUILD_SHARED_LIBS=ON) :

302.120 sec Evolution/Executables/GeneralizedHarmonic/EvolveGhBinaryBlackHole.cpp
297.600 sec Evolution/Executables/GrMhd/GhValenciaDivClean/EvolveGhValenciaDivCleanBondiMichel.cpp
295.980 sec Evolution/Executables/GrMhd/GhValenciaDivClean/EvolveGhValenciaDivCleanTovStar.cpp
261.480 sec Evolution/Executables/GrMhd/GhValenciaDivClean/EvolveGhValenciaDivCleanBns.cpp
244.160 sec Evolution/Executables/GeneralizedHarmonic/EvolveGhSingleBlackHole.cpp
241.690 sec src/Evolution/DgSubcell/Matrices.cpp
232.310 sec Elliptic/Executables/Xcts/SolveXcts.cpp
229.270 sec tests/Unit/ControlSystem/Systems/Test_RotScaleTrans.cpp
228.700 sec Evolution/Executables/ScalarTensor/EvolveScalarTensorSingleBlackHole.cpp
216.450 sec tests/Unit/DataStructures/Tensor/Expressions/Test_EvaluateRank3Symmetric.cpp
201.240 sec tests/Unit/DataStructures/Tensor/Expressions/Test_EvaluateRank3NonSymmetric.cpp
190.220 sec Evolution/Executables/GrMhd/ValenciaDivClean/EvolveValenciaDivCleanFishboneMoncriefDisk.cpp
182.650 sec Evolution/Executables/GrMhd/ValenciaDivClean/EvolveValenciaDivCleanBlastWave.cpp
176.700 sec src/NumericalAlgorithms/FiniteDifference/PositivityPreservingAdaptiveOrder.cpp
174.860 sec Evolution/Executables/GeneralizedHarmonic/EvolveGhNoBlackHole3D.cpp
162.280 sec src/Evolution/Systems/CurvedScalarWave/Worldtube/PunctureFieldOrder2.cpp
156.660 sec Evolution/Executables/CurvedScalarWave/EvolveWorldtubeCurvedScalarWaveKerrSchild3D.cpp
155.510 sec Elliptic/Executables/Punctures/SolvePunctures.cpp
154.860 sec tests/Unit/DataStructures/Tensor/Expressions/Test_Evaluate.cpp
145.320 sec src/DataStructures/Tensor/Python/InstantiateTensor3d.cpp
143.680 sec tests/Unit/Elliptic/DiscontinuousGalerkin/SubdomainOperator/Test_SubdomainOperator.cpp
143.500 sec Evolution/Executables/CurvedScalarWave/EvolveCurvedScalarWavePlaneWaveMinkowski3D.cpp
139.890 sec Elliptic/Executables/Elasticity/SolveElasticity3D.cpp
139.580 sec src/DataStructures/Tensor/Python/InstantiateTensor2d.cpp
139.350 sec Evolution/Executables/NewtonianEuler/EvolveNewtonianEulerRiemannProblem3D.cpp
134.980 sec src/DataStructures/Tensor/Python/InstantiateTensor1d.cpp
134.620 sec tests/Unit/NumericalAlgorithms/FiniteDifference/Test_PositivityPreservingAdaptiveOrder.cpp
131.430 sec tests/Unit/DataStructures/Tensor/Expressions/Test_Product.cpp
130.970 sec Evolution/Executables/ScalarWave/EvolveScalarWave3D.cpp
130.800 sec Elliptic/Executables/Poisson/SolvePoisson3D.cpp
130.740 sec tests/Unit/DataStructures/Tensor/Expressions/Test_MixedOperations.cpp
129.440 sec Evolution/Executables/ForceFree/EvolveForceFree.cpp
128.690 sec Evolution/Executables/RelativisticEuler/Valencia/EvolveValenciaSmoothFlow3D.cpp
126.710 sec Evolution/Executables/RadiationTransport/M1Grey/EvolveM1GreyConstantM1.cpp
125.200 sec tests/Unit/Domain/Creators/Test_Sphere.cpp
122.980 sec Evolution/Executables/GeneralizedHarmonic/EvolveGhNoBlackHole1D.cpp
122.450 sec Evolution/Executables/Cce/CharacteristicExtract.cpp
119.130 sec Evolution/Executables/NewtonianEuler/EvolveNewtonianEulerRiemannProblem2D.cpp
118.080 sec Evolution/Executables/Cce/AnalyticTestCharacteristicExtract.cpp
116.190 sec Evolution/Executables/NewtonianEuler/EvolveNewtonianEulerRiemannProblem1D.cpp
115.460 sec Elliptic/Executables/Elasticity/SolveElasticity2D.cpp
112.950 sec src/Evolution/Systems/Ccz4/Ricci3.cpp
112.250 sec tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateCons3d.cpp
111.940 sec Evolution/Executables/CurvedScalarWave/EvolveCurvedScalarWavePlaneWaveMinkowski2D.cpp
111.070 sec src/NumericalAlgorithms/Spectral/Projection.cpp
110.660 sec tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateMixed3d.cpp
110.280 sec Evolution/Executables/ScalarWave/EvolveScalarWave2D.cpp
110.200 sec Evolution/Executables/Burgers/EvolveBurgers.cpp
109.940 sec Evolution/Executables/RelativisticEuler/Valencia/EvolveValenciaSmoothFlow2D.cpp
109.900 sec tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateMixed2d.cpp
109.020 sec Elliptic/Executables/Poisson/SolvePoisson2D.cpp
108.830 sec tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateCons2d.cpp
108.440 sec Evolution/Executables/CurvedScalarWave/EvolveCurvedScalarWavePlaneWaveMinkowski1D.cpp
108.280 sec Evolution/Executables/ScalarAdvection/EvolveScalarAdvection2D.cpp
107.820 sec tests/Unit/Elliptic/Systems/Xcts/Test_Equations.cpp
107.270 sec Elliptic/Executables/Poisson/SolvePoisson1D.cpp
105.960 sec Evolution/Executables/RelativisticEuler/Valencia/EvolveValenciaSmoothFlow1D.cpp
105.660 sec Evolution/Executables/ScalarAdvection/EvolveScalarAdvection1D.cpp
105.600 sec src/NumericalAlgorithms/Spectral/SwshDerivatives.cpp
105.300 sec tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateCons1d.cpp
104.650 sec tests/Unit/Elliptic/DiscontinuousGalerkin/Test_DgOperator.cpp
104.140 sec Evolution/Executables/ScalarWave/EvolveScalarWave1D.cpp
104.060 sec tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateMixed1d.cpp
103.900 sec tests/Unit/Domain/Creators/TimeDependence/Test_UniformTranslation.cpp
 99.810 sec tests/Unit/Evolution/DiscontinuousGalerkin/Actions/Test_ApplyBoundaryCorrections.cpp
 97.630 sec src/Domain/Creators/RegisterDerivedWithCharm.cpp
 92.850 sec tests/Unit/ParallelAlgorithms/LinearSolver/Multigrid/Test_MultigridPreconditionedGmresAlgorithm.cpp
 92.690 sec tests/Unit/Domain/CoordinateMaps/Test_CoordinateMap.cpp
 92.440 sec tests/Unit/Domain/Creators/Test_BinaryCompactObject.cpp
 91.240 sec Executables/ExportCoordinates/ExportTimeDependentCoordinates3D.cpp

8750.080 MB Evolution/Executables/GeneralizedHarmonic/EvolveGhBinaryBlackHole.cpp
8722.072 MB Evolution/Executables/GrMhd/GhValenciaDivClean/EvolveGhValenciaDivCleanBondiMichel.cpp
8420.084 MB Elliptic/Executables/Xcts/SolveXcts.cpp
8031.824 MB Evolution/Executables/GrMhd/GhValenciaDivClean/EvolveGhValenciaDivCleanBns.cpp
7920.492 MB Evolution/Executables/GrMhd/GhValenciaDivClean/EvolveGhValenciaDivCleanTovStar.cpp
7342.564 MB Evolution/Executables/ScalarTensor/EvolveScalarTensorSingleBlackHole.cpp
6922.912 MB Evolution/Executables/GeneralizedHarmonic/EvolveGhSingleBlackHole.cpp
6734.836 MB Evolution/Executables/GrMhd/ValenciaDivClean/EvolveValenciaDivCleanFishboneMoncriefDisk.cpp
6261.100 MB Evolution/Executables/Cce/AnalyticTestCharacteristicExtract.cpp
5985.876 MB Evolution/Executables/GrMhd/ValenciaDivClean/EvolveValenciaDivCleanBlastWave.cpp
5671.680 MB Evolution/Executables/Cce/CharacteristicExtract.cpp
5580.808 MB Evolution/Executables/GeneralizedHarmonic/EvolveGhNoBlackHole3D.cpp
5265.368 MB Evolution/Executables/CurvedScalarWave/EvolveWorldtubeCurvedScalarWaveKerrSchild3D.cpp
5095.824 MB Elliptic/Executables/Punctures/SolvePunctures.cpp
4950.024 MB Elliptic/Executables/Elasticity/SolveElasticity3D.cpp
4911.736 MB Evolution/Executables/CurvedScalarWave/EvolveCurvedScalarWavePlaneWaveMinkowski3D.cpp
4876.120 MB Evolution/Executables/NewtonianEuler/EvolveNewtonianEulerRiemannProblem3D.cpp
4574.692 MB Elliptic/Executables/Poisson/SolvePoisson3D.cpp
4382.796 MB Evolution/Executables/ForceFree/EvolveForceFree.cpp
4299.268 MB Evolution/Executables/GeneralizedHarmonic/EvolveGhNoBlackHole1D.cpp
4289.500 MB Evolution/Executables/RelativisticEuler/Valencia/EvolveValenciaSmoothFlow3D.cpp
4072.756 MB Evolution/Executables/ScalarWave/EvolveScalarWave2D.cpp
4066.812 MB Evolution/Executables/ScalarWave/EvolveScalarWave1D.cpp
4005.256 MB Evolution/Executables/ScalarWave/EvolveScalarWave3D.cpp
4004.044 MB Evolution/Executables/CurvedScalarWave/EvolveCurvedScalarWavePlaneWaveMinkowski1D.cpp
3996.524 MB Evolution/Executables/CurvedScalarWave/EvolveCurvedScalarWavePlaneWaveMinkowski2D.cpp
3957.028 MB Evolution/Executables/Burgers/EvolveBurgers.cpp
3949.196 MB Evolution/Executables/RadiationTransport/M1Grey/EvolveM1GreyConstantM1.cpp
3945.764 MB Evolution/Executables/RelativisticEuler/Valencia/EvolveValenciaSmoothFlow1D.cpp
3942.200 MB Elliptic/Executables/Elasticity/SolveElasticity2D.cpp
3937.796 MB Evolution/Executables/RelativisticEuler/Valencia/EvolveValenciaSmoothFlow2D.cpp
3891.008 MB Evolution/Executables/NewtonianEuler/EvolveNewtonianEulerRiemannProblem2D.cpp
3867.144 MB Evolution/Executables/ScalarAdvection/EvolveScalarAdvection1D.cpp
3851.508 MB Evolution/Executables/ScalarAdvection/EvolveScalarAdvection2D.cpp
3840.812 MB Evolution/Executables/NewtonianEuler/EvolveNewtonianEulerRiemannProblem1D.cpp
3836.164 MB tests/Unit/ControlSystem/Systems/Test_RotScaleTrans.cpp
3666.844 MB Elliptic/Executables/Poisson/SolvePoisson2D.cpp
3623.968 MB Elliptic/Executables/Poisson/SolvePoisson1D.cpp
3597.464 MB tests/Unit/Elliptic/DiscontinuousGalerkin/SubdomainOperator/Test_SubdomainOperator.cpp
3163.904 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateMixed2d.cpp
3160.016 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateCons2d.cpp
3069.960 MB Executables/ExportCoordinates/ExportTimeDependentCoordinates3D.cpp
3059.552 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateCons3d.cpp
3058.620 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateMixed3d.cpp
2994.104 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_CharacteristicEvolutionBondiCalculations.cpp
2942.888 MB Executables/ExportCoordinates/ExportCoordinates3D.cpp
2923.024 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateCons1d.cpp
2922.664 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateMixed1d.cpp
2919.164 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_ScriObserveInterpolated.cpp
2898.768 MB Executables/ExportCoordinates/ExportCoordinates2D.cpp
2812.780 MB Executables/ExportCoordinates/ExportCoordinates1D.cpp
2804.504 MB Executables/FindHorizons/FindHorizons3D.cpp
2764.180 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_H5BoundaryCommunication.cpp
2751.484 MB tests/Unit/Evolution/DiscontinuousGalerkin/Actions/Test_ApplyBoundaryCorrections.cpp
2748.080 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_GhBoundaryCommunication.cpp
2629.668 MB tests/Unit/ParallelAlgorithms/ApparentHorizonFinder/Test_ApparentHorizonFinder.cpp
2618.420 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_InitializeCharacteristicEvolution.cpp
2596.344 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_InsertInterpolationScriData.cpp
2542.432 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_AnalyticBoundaryCommunication.cpp
2451.028 MB tests/Unit/ParallelAlgorithms/LinearSolver/Multigrid/Test_MultigridPreconditionedGmresAlgorithm.cpp
2435.640 MB src/DataStructures/Tensor/Python/InstantiateTensor1d.cpp
2428.016 MB tests/Unit/Evolution/Systems/Cce/Actions/Test_RequestBoundaryData.cpp
2421.632 MB src/DataStructures/Tensor/Python/InstantiateTensor2d.cpp
2395.324 MB src/DataStructures/Tensor/Python/InstantiateTensor3d.cpp
2300.408 MB tests/Unit/PointwiseFunctions/GeneralRelativity/Test_IndexManipulation.cpp
2286.588 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateNoncons3d.cpp
2268.028 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateNoncons2d.cpp
2244.228 MB IO/Importers/Test_VolumeDataReaderAlgorithm3D.cpp
2211.032 MB tests/Unit/Helpers/Evolution/DiscontinuousGalerkin/Actions/InstantiateNoncons1d.cpp
2192.392 MB tests/Unit/Evolution/Systems/Cce/Test_OptionTags.cpp
2160.480 MB tests/Unit/Elliptic/Systems/Xcts/Test_Equations.cpp
2146.952 MB tests/Unit/Evolution/Systems/Cce/Test_SwshDerivatives.cpp
2136.520 MB tests/Unit/ControlSystem/Systems/Test_Translation.cpp
2133.984 MB tests/Unit/ControlSystem/Systems/Test_Expansion.cpp
2125.880 MB tests/Unit/ControlSystem/Systems/Test_Rotation.cpp
2096.392 MB tests/Unit/ControlSystem/Systems/Test_Shape.cpp
2094.124 MB tests/Unit/ParallelAlgorithms/LinearSolver/Multigrid/Test_MultigridAlgorithm.cpp
2041.284 MB src/NumericalAlgorithms/FiniteDifference/PositivityPreservingAdaptiveOrder.cpp
2025.700 MB tests/Unit/Evolution/Initialization/Test_SetVariables.cpp
2020.072 MB tests/Unit/Elliptic/DiscontinuousGalerkin/Test_DgOperator.cpp
2019.016 MB tests/Unit/ParallelAlgorithms/LinearSolver/Gmres/Test_DistributedGmresPreconditionedAlgorithm.cpp
2016.648 MB tests/Unit/PointwiseFunctions/GeneralRelativity/Test_ComputeGhQuantities.cpp
2009.544 MB tests/Unit/DataStructures/Tensor/Expressions/Test_Evaluate.cpp
knelli2 commented 1 year ago

An update from @macedo22 on her exploration of the BBH executable:

The string literals for all the options help text seem to cost ~1GB of compile RAM and ~2min of compile time. By making all the literals just be empty, it saved this much (see commit). I also tried changing the Options::String type to then be a std::string and that compiled so I haven’t found any evidence yet that there is some current compilation requirement that they be constexpr.

macedo22 commented 11 months ago

Exploration of code changes and impact on compile time and memory for compiling EvolveGhBinaryBlackHole: link. See this sheet of that document for 11/09/23 tech call update on areas of the code that, as of now, seem to be big contributors to compile time RAM.

nilsvu commented 6 months ago

About option help strings: would it help to convert them to static functions, like static std::string help() { return "help"; }? That would be helpful anyway, because if constexpr can be used in them.

wthrowe commented 6 months ago

I think the help string thing ended up being a red herring. The test that showed improvement accidentally introduced a bunch of undefined behavior, which was probably the real cause.

macedo22 commented 6 months ago

@nilsvu I tried various versions of this back in the fall with clang-13 Release and the compile time and peak RAM either got worse or was about the same. I just repeated this for gcc-9 Release and am unfortunately seeing no difference in the compile time or peak RAM.