Provide Socrates executables

[lsoucasse]

Currently, compilation of the socrates code is mandatory. We would like to simplify this step by directly providing executables to download for different OS. Two executables are used by JANUS prep_spec and Cl_run_cdf. We might however change this and use the socrates library instead (radlib.a), as it is done in AGNI.

[stefsmeets]

I set up a workflow like this a couple of years ago: https://github.com/hpgem/tetgen/blob/main/.github/workflows/build_linux.yml

This builds the code and makes the binaries availabe via a github release automatically. We could quite easily replicate that for SOCRATES. Zip the build output and make it available.

The tricky part, imo, is that SOCRATES is not a single binary. It is unclear to me whether you can just dump all the binaries in your system path and it will just work. Also, how do we deal with all the environment variables that SOCRATES seems to require (via set_rad_env).

[stefsmeets]

Just had a quick test with this by putting the suggested executables on my path. Still run into issues with missing files.

Additional executables/libraries:

• fmove (required by Cl_run_cdf)
• currentsw.* (netCDF files produced by SOCRATES, required by radCompSoc)
• ...?

It seems that the currentsw files are generated by socrates, but renamed by janus using fmove.

https://github.com/FormingWorlds/JANUS/blob/f53287a04f991eb3a3b12f3adb7a4a983466f8d0/src/janus/utils/socrates.py#L166-L170

fmove itself is a very simple bash script to rename files that might as well be python: https://github.com/FormingWorlds/SOCRATES/blob/main/sbin/fmove

fmove seems to always succeed, even if it does nothing. Would be better if it crashed if the expected file is not found.

[stefsmeets]

Oh, wow, so not only is Cl_run_cdf a hugely compex bash script that not only depends on other tools within socrates, it also directly reads the source code to extract parameters. 🧐

https://github.com/FormingWorlds/SOCRATES/blob/5f588e6518af2a3f6610de09348211b192e4b35f/sbin/Cl_run_cdf#L12-L14

This script parses the fortran source code.

I think this makes it quite complex to package this nicely for distribution.

[timlichtenberg]

Maybe @nichollsh can comment on the implementation in AGNI? Perhaps instead of going through with this, implementing a Python version of accessing radlib.a, like done in the Julia implementation, would be the solution?

[stefsmeets]

The nice thing about using the executables as is, is that it will not require many changes on the JANUS side. Accessing radlib.a comes with its own set of challenges.

I could see we end up with some sort of tool that (depending on the platform) downloads the entire compiled distribution and simply calls that. I think we can set this up in CI quite earily.

For portability, then the only thing we need is to replicate is set_rad_env on the python side, before calling any socrates code.

[stefsmeets]

AGNI also seems to rely on the user compiling SOCRATES by themselves, so it does not solve the compilation/installation problem: see Installation via get_socrates.sh

[stefsmeets]

Here is how I see it at the moment:

• [x] Set up CI to build SOCRATES automatically on every commit. -> Already [exists].(https://github.com/FormingWorlds/SOCRATES/blob/main/.github/workflows/build.yml)
• [x] Adapt the workflow above to make the binaries available via github releases -> https://github.com/nichollsh/SOCRATES/pull/6
• [ ] Make first release available on SOCRATES releases page
• [x] Add download tool to pull SOCRATES release to user directory
• [x] Have a thin wrapper in JANUS to set up the environment for socrates (mimicking set_rad_env)

This requires minimal adaptation on the JANUS side. And I believe the SOCRATES code is mirrored from elsewhere, so this keeps the changes on the SOCRATES side also minimal.

[nichollsh]

AGNI does rely on the libSOCRATES.so library which the user has to compile themselves, at the moment, but it doesn't rely on Cl_run_cdf.

Why does set_rad_env need to be rewritten? It should be quite portable since it's a shell script.

And yes, SOCRATES is mirrored from the Met Office SCM repository which requires an account to access. They will be moving to GitHub in the coming years, but it would still be good to keep changes to SOCRATES as additive to make things smoother.

[stefsmeets]

Why does set_rad_env need to be rewritten? It should be quite portable since it's a shell script.

It has a hard-coded path in the script: RAD_DIR=/home/stef/python/socrates. Either we update it or dynamically set the environment variables in Python.

[nichollsh]

Ah I see. This path is set when the model is compiled, so if we are intending to distribute executables then it wouldn't be updated appropriately.

I think that setting environment variables makes sense, since that's mostly what this script is doing anyway.

[stefsmeets]

I'm testing this in https://github.com/FormingWorlds/JANUS/blob/socrates/src/janus/socrates.py

Seems to work well enough to pass the tests. Which means that we can put socrates anywhere we want and be able load it without having to source set_rad_env.

[lsoucasse]

If the external build solution does not work, we can simply provide a "install socrates" command (like the download data command) that downloads the source and runs the build script.

By the way, there is already a socrates.py file in janus/utils.

[stefsmeets]

I changed the function to instead download the latest tarball and compile it. This seems to work quite well.

[stefsmeets]

Having static binaries available for SOCRATES depends on static libraries for netcdf. These are not available by default on Ubuntu, and this Github Actions. See https://bugs.launchpad.net/ubuntu/+source/netcdf/+bug/1698368

Credits to our colleague @v1kko to help debug this