Add Windows and OSX to CI

[bobturneruk]

We currently support Linux using a Conda spec-file (which is OS specific) to define an environment for execution. This offers, in principle, a highly reproducible execution environment, increasing the chances of model runs producing the same output on different systems. This means all of our CI testing is done in this one environment. However, development takes place on Linux, Windows and OSX. This means sometimes tests fail on Windows as they are not caught by CI. Should we:

1 Keep it as it is? 2 Add support for Windows and OSX in CI? 3 Make it clear that Windows and OSX are not supported and move all development to Linux? 4 Move from an OS-specific spec-file to a cross-platform environment.yml file? 5 Provide a Docker container for reproducible runs? 6 Do something else?

[github-actions[bot]]

Heads up @magicicada @bobturneruk @aflag @WPettersson @alex-konovalov @may1066 @mrow84 - the "Simple Network Sim" label was applied to this issue.

[bobturneruk]

I'd be OK with 3. I'd be OK with 2, 4 and 5, as I think the container will give back most of what we'd lose from going from the spec-file to the environment.yml.

[aflag]

I think we can add windows and osx support using the environment.yml file. So, linux would be the system we are able to pin all versions (and hopefully the system where production runs will be executed), but we'd also test that it works in windows and mac os x too. If we use the spec-file in linux but not on the other two systems, it should be easy to see when tests are broken due to different dependency versions.

[magicicada]

If 4 is convenient, then I like it. If not, then 3 is probably acceptable for the work of SCRC, though would really limit future use (and on a personal note I'm not currently running a Linux machine, strictly speaking)

[bobturneruk]

I think we could have the CI do:

	Linux	OSX	Windows 10
spec-file	✔️	❌	❌
environment.yml	✔️	✔️	✔️

Then if we containerise, we do so with the spec-file.

[bobturneruk]

Great - I think this should keep everyone happy!

[widdowquinn]

I think 2, 4 and 5 are all workable solutions. If I had to pick one, I'd go with (2). On Biopython we test for Linux on Travis, and Windows on Appveyor. I believe CircleCI and Appveyor would both be able to handle tests across all three OSes.

[bobturneruk]

Thanks @widdowquinn! Do you use spec-files for each OS? (GitHub actions will also do all OSs.)

[widdowquinn]

On Biopython, there are separate config files for Travis and Appveyor. The Appveyor config uses a conda environment, the Travis environment handles some dependencies through Tox. Both directly install named dependencies in the CI config files, rather than using spec-files. Dependencies are not named in setup.py.

It's not the most elegant solution.

[olexandr-konovalov]

GAP also has separate config files for Travis (Linux and macOS) and Appveyor (Windows).

[aflag]

I think it should be possible to create spec-files for all OSes, but it's not entirely trivial. Maybe there's some tool for that. I think we need to discuss how the "official run" will happen. That should dictate which OS gets the spec-file.

[bobturneruk]

I think we should have:

• One spec-file.txt which is the 'official' execution environment for reproducible runs and will dictate which OS these should happen on (almost certainly Linux if OK with @magicicada?).
• One cross-platform environment.yml which is a less highly specified environment for development.

CI (probably multiple providers) should cover all platforms using environment.yml and the 'official' one with spec-file.txt (which is the current setup). I'm increasingly inclined to say we should also wrap the spec-file approach in a docker container to provide cross-platform reproducible runs including on HPC via singularity.

[bobturneruk]

Having spoken to colleagues, my strong preference is to use GitHub actions for Windows 10 and OSX CI.