Build and distribute ASCOT5 via conda

[miekkasarki]

1. Write a conda recipe that builds ascot5_main (and other binaries), libascot.so and installs a5py along with any dependencies.

   • This requires that path to libascot.so is not hard-coded (but I guess we want to retain hard-coded path when using ascot from source or not using it in conda)
   • Any non-code files listed in MANIFEST.IN and accessed via setuptools during runtime.
   • All libraries should be dynamically linked

2. Test that the resulting tarball can be used to install ASCOT5 in other platforms (we can probably ignore Windows for now).

3. GitHub action to build the tarball (but not upload it except as a temporary artifact).

4. Make a PR to conda-forge to share the recipe and distribute via official channels.

[miekkasarki]

The first bullet is now mostly done, but the second bullet fails (a5py is not installed and complains about GLIBC version mismatch). However, the main binary and libascot.so seem to work otherwise. The magic happens mostly in meta.yaml and build.sh, but also setup.py and MANIFEST.IN required modifications. However, I guess we should also switch from setup.py to pyproject.toml.

Note: I have very little idea what I'm doing so feel free to point any errors.

@sjjamsa tagging you in case this issue is relevant to your interests.

[miekkasarki]

Think I solved the issue with a5py (it was due to setuptools find_package) and the second issue could(?) be solved by building the package with the oldest possible GLIBC.

[miekkasarki]

Managed to build ascot5 on target platform! 🎉

Now to deal with all meta.yamls, setup.pys, env.yamls, I think we should define supported installations or installation instructions. I propose the following:

Installing

Here are the three supported ways to install ASCOT5. The easiest way is to install the code in Conda environment but, since ASCOT5 is a high-performance code, the code will operate with a (significantly) reduced performance.

Therefore the recommended way is to do the "HPC installation" to compile the executables (e.g. ascot5_main) on the cluster where you run the simulations, and then do the "Full installation" separately on the same cluster or somewhere else where you plan to do the pre- and post-processing. If you don't care about the performance, just do the "Full installation" and be done with it.

Full installation

Installs all the dependencies, builds the code (with reduced performance) and the associated library libascot.so, and installs the Python pre- and post-processing package a5py with all features available.

First install miniconda/Anaconda, or activate the relevant module in your system, and then type:

(Once ASCOT5 is available on conda-forge)

conda env create -n ascot-env
conda activate ascot-env
conda install -c conda-forge ascot5

The current way (requires building from source) and also one that will be used in CI/CD:

git clone https://github.com/ascot4fusion/ascot5.git
conda env create -f=ascot5/environment.yaml
cd ascot5
make ascot5_main
make libascot
pip install -e .

HPC installation

Builds binaries from the source to optimize it for the current platform. Does not build libascot.so and installs only the minimum features of a5py to edit options and manipulate the HDF5 file. Even installing the a5py package is optional if you only submit the jobs on this platform.

<do here any module loads and export statements that are platform specific>
git clone https://github.com/ascot4fusion/ascot5.git
cd ascot5
make ascot5_main <settings for the compiler/make>

# Install a5py in a virtual environment
cd ..
virtualenv -p python3 --system-site-packages ascot-env
source ascotenv/bin/activate
pip install -e ascot5/a5py

Developer installation

This makes the full installation while also including all the additional tools to build the documentation.

git clone https://github.com/ascot4fusion/ascot5.git
conda env create -f=ascot5/environment-dev.yaml
cd ascot5
make ascot5_main
make libascot
pip install -e .

[miekkasarki]

In the end we would have (files):

meta.yaml contains all full installation (but not developer) dependencies. Will be moved to a feed-stock repo.

setup.py will be just a stub replaced by pyproject.toml which should have the minimum dependencies needed by a5py.

environment.yaml has all the full installation (but not developer) dependencies.

environment-dev.yaml has all possible dependecies.

[miekkasarki]

Ok, now all the above mentioned installation ways are implemented. The next step would be to update the Docker so that these will be tested as part of CI and conda environment will be used when compiling docs & running tests.

Adding some TODOs:

• [x] Logo and icon were moved inside a5py and this should be taken into account when building docs.
• [x] Try to single-source the version number.

[miekkasarki]

To continue this work, I'll make a new branch containing all changes to this repository and workflows that use Conda. This branch will be merged to main.

The recipe is moved to my personal fork (didn't bother to include it here since it is a temporary fork) of conda-forge https://github.com/miekkasarki/staged-recipes/tree/main

[miekkasarki]

Accidentally closed by PR.

[miekkasarki]

I managed to compile and run with MPI on a cluster both using the native libraries and using the ones conda provided. This makes ASCOT5 fully compatible with conda so now it's all about finishing the recipe and pushing it to conda-forge.