We should specify modules per branch/realisation

[SeanBryan51]

We should be able to run multiple branches that have conflicting modules/dependencies. For example, in https://github.com/CABLE-LSM/bench_example_GW, the ground water branch is compiled with intel-compiler/2019.5.281 and netcdf/4.6.3 but when we run the tests, we load modules for intel-compiler/2021.1.1 and netcdf/4.7.4 globally across all realisations. We should instead specify and load modules per branch so that the correct modules can be loaded for each branch. This would require 'module load' statements to be executed by benchcab internally instead of the job script (which loads modules globally for all branches).

Edit: after the discussion below, it is unlikely we need to specify modules per branch/realisation. But we should overwrite the modules in the custom build script so that we are using the same modules when compiling and running CABLE.

TODO:

• [ ] remove the custom_build() method in repository.py
• [ ] we should add documentation on what benchcab is doing to build CABLE. For example, how are modules are loaded from benchcab (for example the build script should not contain lines like this: module load gcc && some_cmd otherwise some_cmd will not be executed).
• [ ] we should state in the documentation for the build_script config parameter that it "describes the path to a shell script".

[ccarouge]

I don't think we should specify modules per branch. The goal is for all the branches to be compiled with the exact same versions of the modules/dependencies. We don't want to test for differences coming from the dependencies. For the moment, the custom build pathway relies on the user to set the modules exactly the same as what is specified in the config.yaml file. So the issue is on relying on the build script to load the modules. Maybe the best would be to change the custom build pathway. Instead of using the build script as is, we should remove all 'module load/add' statements from the script and load the modules from benchcab. Typically, the same as the normal build but with a different script. I think now that would be a very simple change to implement.

[SeanBryan51]

I don't think we should specify modules per branch. The goal is for all the branches to be compiled with the exact same versions of the modules/dependencies.

Yes I agree, we should be using the same dependencies for when we are accepting new code submissions. This issue is more directed towards the use case of when we want to compare say the latest trunk with a legacy branch of CABLE which requires a custom build script for compilation. In this case, the user should be able to specify modules per branch. For example, we could run into the scenario where the legacy branch requires modules that are not compatible with the trunk.

Maybe the best would be to change the custom build pathway. Instead of using the build script as is, we should remove all 'module load/add' statements from the script and load the modules from benchcab.

I think this is a good idea. I find it a bit unsettling that the custom build script is loading any modules it likes to compile and when we running CABLE we are loading modules that are not necessarily the same ones that were used in compilation.

Although, I'm still convinced that being able to specify modules per branch (at least as an optional config parameter) would be useful due to the possibility of having conflicting dependencies between legacy branches.

[ccarouge]

Although, I'm still convinced that being able to specify modules per branch (at least as an optional config parameter) would be useful due to the possibility of having conflicting dependencies between legacy branches.

I would be very surprised if this was a problem with CABLE offline. It might happen when we look at running benchcab for AMIP simulations. But then the compilation and submit a simulation will be different. So I would only worry about that when we reach that point and see what is needed. For now, I would simply use the same pathway for the custom build script as for the default build.