Unfortunately we have been mostly discussing this in the meetings and in Slack, so there is no permanent record of the development process here. I'll try to summarize the main points:
We develop ASCOT5 so that all inputs can be initialized from dictionaries using ctypes, and likewise all output is accessed via ctypes. This is because in IMAS the input will not be stored in HDF5 and storing the output to the HDF5 must be optional. This is accomplished by using the VirtualRun infrastructure and libproviders.py.
Up-to-date version of ASCOT5 must be maintained at EuroFusion Gateway. There will be a modulefile that must be loaded in order to use ASCOT5 actor.
ASCOT5 actor itself will be a Python script that uses libascot to run the simulation (and for pre- and postprocessing). There will be no C-interface between IMAS and ASCOT5 and this means that we have to handle MPI within the Python script using mpi4py. The Python script (or wrapper as we call it) must spawn the processes as mpirun (or similar) is not used.
The simulation options must be available in xml format so that they may be changed in IMAS.
Mostly done. We still need to fully develop the interface between ASCOT5 dictionaries & output and IMAS IDS.
Mostly done.
We have a proof of principle (copy-pasted at the end of this message).
Not done yet.
testmpi2.py
#!/usr/bin/env python
from mpi4py import MPI
import numpy as np
import sys
import pickle
from a5py import Ascot
if __name__ == "__main__":
# Prepare data
a5 = Ascot("ascot.h5", mute="yes", create=True)
a5.data.create_input("bfield analytical iter circular")
# Spawn MPI processes
comm = MPI.COMM_SELF.Spawn(
sys.executable,
args=['-c', 'import testmpi2; testmpi2.worker()'],
maxprocs=2)
comm.Disconnect()
def worker():
# This function is run in each MPI process
comm = MPI.Comm.Get_parent()
size = comm.Get_size()
rank = comm.Get_rank()
# Init Ascot object in each process but only root reads the data
a5 = Ascot("ascot.h5")
data = None
if rank == 0:
data = a5.data.bfield.active.read()
data = MPI.COMM_WORLD.bcast(data, root=0)
# Initialize data on each process
a5.input_init(bfield=data)
comm.Disconnect()
ASCOT5 should be an IMAS actor or so.
The ITER The Integrated Modelling & Analysis Suite (IMAS)