Ensure conda environment covers what is needed for METplus

[jfrost-mo]

As METplus will be run as part of CSET, we need CSET's environment to provide everything that METplus needs.

@JorgeBornemann when you talk about the METplus environment, does that just need to be what is required to run METplus, or is that everything required to compile and build it too?

Issue transferred from https://github.com/MetOffice/CSET-workflow/issues/76.

[jfrost-mo]

Ideally the build-conda task could be repurposed to build the conda environment with al the METPlus requirements, activate the environment and build MET/METPlus on it. All the MET/METPlus environment variables will be the same for building and using. In terms of sourcing the environment via modules, there has to be a single module or ensure that the combination of modules don't interfere with each other.

Original comment by JorgeBornemann

[jfrost-mo]

Ideally the build-conda task could be repurposed to build the conda environment with al the METPlus requirements, activate the environment and build MET/METPlus on it.

If we are going to rebuild MET anyway, might it make sense to actually package it on conda-forge? That would give us separation of concerns between building it, and using it, and would mean that we can avoid pulling in some fairly hefty dependencies in terms of compilers, etc.

Original comment by jfrost-mo

[jfrost-mo]

I am not familiar with packaging on conda-forge. But I presume that we have to check with the MET development team (DTC), as we would be putting their code in a public repo. If that is not a problem and the packaging in coda forge implies that in any platform it can be brought back with a conda install, then I am allfor it.

Original comment by JorgeBornemann

[jfrost-mo]

Its not an issue from a legal standpoint (MET and METplus are Apache-2.0 licenced, permitting redistribution), though it would definitely be polite to ask. This is an issue open to do it, so it seems like something that may be of interest to MET. https://github.com/dtcenter/MET/issues/1606

Original comment by jfrost-mo

[jfrost-mo]

Some guidance around using a container. I don't think we will go this route, but I'm putting it here so all our options are in one place.

The good news is that it's probably technically possible to convert DTC's docker images into functioning Singularity images. The trickier bit is dealing with the cyber security questions that would allow use of those images on VDI and SPICE. For example, if DTC were notified of a compromised docker image, do they have a process that would allow them to notify us? Are there well understood and appropriate controls on the build processes that create the DTC images? Who in the Met Office is going to own the Singularity images and the process for creating them?

The AVD team have been through similar processes, with the help of the cyber security team, to evaluate the conda distributions they make available on Artifactory. For example: https://metoffice.atlassian.net/wiki/spaces/AVD/pages/3498934295/Anaconda+vs+Conda-forge

- Richard Hattersley

Original comment by jfrost-mo

[jfrost-mo]

Thanks, it is useful information to have, and maybe one day we will use it, but uptake of containerisation is quite varied across institutions at the moment, so it will hinder portability, rather then favouring it.

Original comment by JorgeBornemann

[John-Sharples]

Having MET/METplus available via conda-forge would be valuable to the Bureau of Meteorology too. Please reach out if you think this is something we could collaborate on.

[jfrost-mo]

The main thing needed for this is a reliable and portable build script for MET and its dependencies. My experience with C++ build systems is limited, and I've not managed to get it working, but if we can get that it can be converted into a conda package.