Repo organisation

[johnomotani]

I'm planning to set up BOUT++ on ARCHER2 for a few users, am thinking it would be nice to add the config(s) here. Wondering how it's best to organise all the different configs.

One possibility that comes to mind would be to switch to having a different branch for each machine (or possibly for each group of machines). Pros:

• No clutter of configs for machines that you aren't using.
• If we want to include the BOUT-dev repo as a submodule with a known-good version for each machine config, then only one copy of BOUT-dev will exist for each branch. If we have all machines as subdirectories, then potentially each one wants a different version of BOUT-dev, so you could end up with many copies of BOUT-dev. Cons:
• Clutter of branches - user needs to know or be told which branch to use. Hopefully "machine name" -> "branch name" is a simple solution to this though!

Any thoughts, suggestions, other alternatives?

[johnomotani]

There's now an 'archer' branch.

It could work to have main contain just a README, and all the actual scripts, etc. in machine-specific branches?

[ZedThree]

I'm undecided! Having a flat structure with one branch would make things more discoverable, but having a pinned version of BOUT-dev as a submodule in each branch is quite appealing.

Also, I wonder if we should start trying to use CMake presets for common build configurations, and toolchains for machine-specific configurations. I think at least the LLNL config scripts in this repo also encompass setting up the third party libraries we use, which are not really what presets/toolchains are for (as far as I understand things). Those could be set up with a "super-project" CMakeLists.txt, which would also make propagating the dependencies much easier.

[johnomotani]

LLNL config scripts in this repo also encompass setting up the third party libraries we use

For the archer config also, I'm downloading/compiling PETSc (because I couldn't get the PETSc in ARCHER2's module to link correctly).

[johnomotani]

I had to set up BOUT++ on a new laptop, so there's now a generic-4.4 branch too, which aims to get BOUT++ and dependencies compiled on a standard Linux machine (tested on Linux Mint 20.2) with just a couple of commands (very similar to the archer setup).

[johnomotani]

There is now also a marconi branch, with configuration for gnu compilers with openmpi, intelmpi (2020 version), or intelmpi2018. I did not manage to get Intel compilers working (see #3).

[johnomotani]

Just to note, whatever we decide to do to resolve this, we should update the README.md on main to document it, as that is what most users will see as the front-page of BOUT-configs.

[ZedThree]

I would prefer a single branch, maybe with a top level machines directory to keep it from getting cluttered. That would make PR, for instance, easier to manage.

[johnomotani]

Definitely advantages to having everything in a single branch.

If we go the single-branch route, how are we managing the version of BOUT-dev that goes with each config?

• We wouldn't want to have BOUT-dev as a submodule in each subdirectory (that would waste disk space and bandwidth).
• For the stable releases, it could make sense to have BOUT-dev as a submodule at the top level, and then each machine-specific config can compile that.
  • If we want to support next too (maybe in a separate branch?) I'm not so sure it's good to have a shared version of BOUT-dev - there's a significant possibility of updates to next breaking things, so you'd want the BOUT-configs maintainer for each machine to check that the version compiles and passes tests, and to tweak the configs if necessary. That'd probably be annoying to coordinate across several machines (how would we decide when to move to a new version of next?). If the different machines had separate branches, we avoid that issue as the maintainer for each machine decides independently when to update (when there's demand for it).
• We could not include BOUT-dev as a submodule at all, and add a line in each configuration to clone the repo. This has some advantages - I took this approach with STORM in my STORM-configs repo because it decouples the version of the code from the version of the configs - but unless we add some extra tricks, it means the user has to update BOUT-configs and BOUT-dev separately. The submodule approach is nice because there's just one place to go - you go to the BOUT-configs directory git pull; ./<build script>.
  • Having done both, I think having BOUT-dev as a submodule is better for users who don't want to modify BOUT themselves, and cloning BOUT-dev but not tracking it with BOUT-configs is better for developers who do want to modify BOUT.

Maybe the best of both worlds would be to have the main branch have a submodule with the latest release of BOUT-dev, and configs for next be in (machine-specific) branches which can be merged when BOUT-dev/next becomes the new BOUT-dev/master?

[ggeorgakoudis]

Is this issue relevant anymore?

[johnomotani]

We've stayed so far with 'each machine is a separate branch', but I don't think that was an actual decision...

[ggeorgakoudis]

We decided on a per-folder structure instead of per-branch. We are hooking into spack now for building a reproducible environment across machines. Are you still working on marconi? Can you volunteer to create a PR for a marconi config modeled after the perlmutter one?