pwollstadt / IDTxl

The Information Dynamics Toolkit xl (IDTxl) is a comprehensive software package for efficient inference of networks and their node dynamics from multivariate time series data using information theory.
http://pwollstadt.github.io/IDTxl/
GNU General Public License v3.0
237 stars 76 forks source link

MPI support for IDTxl #80

Closed daehrlich closed 7 months ago

daehrlich commented 2 years ago

Added support for MPI parallelization for serial CPU estimators.

Requires mpi4py>=3.0.0

To use MPI, all one needs to do is add a MPI=True flag in the Estimator settings dictionary (one can optionally provide a n_workers argument) and start the script using

mpiexec -n 1 -usize <max workers + 1> python

on systems with MPI>=2.0 or

mpiexec -n <max workers + 1> python -m mpi4py.futures

on legacy MPI 1 implementations.

Internally, added class MPIEstimator, which is a decorator for arbitrary serial (is_parallel()==False) Estimators sending chunks of data to individual Estimator instances on MPI worker ranks using mpi4py.futures.

Limitations:

Auxiliary changes: