search

nest / nest-simulator

The NEST simulator
http://www.nest-simulator.org
GNU General Public License v2.0
541 stars 367 forks source link

Documentation on nest and multi-processing / multi-threading #1581

Closed terhorstd closed 4 years ago

terhorstd commented 4 years ago

Describe the flaw The simulator is built for parallel processing, but having many nodes in a cluster it is tempting to try to use python's multiprocessing package to speed up also other parts of the simulation scirpts. Information on the possible scenarios, relation to Python's multi-processing API, or multi-threading with other communications libraries is missing.

Suggest an improvement References to a conceptual document describing the mpi/threading environment should be in relevant sections of guides and reference docs.

Supporting materials Was discussed on NEST mailing list

Desktop/Environment relevant for all users of clusters or multi-core hardware.

juanmav commented 4 years ago

@terhorstd I dont know but maybe it could help. Here the sample code that I made to test multiprocessing with nest. https://github.com/juanmav/spiking/blob/master/smalltestcase.py

sarakonradi commented 4 years ago

@juanmav Thanks for the sample code!

@heplesser I remember that, in the mailing list, you emphasized the importance of not using multiprocessing with NEST and that there are currently no scenarios where it is okay to use Python's multiprocessing API, at least not until we have a careful assessment about the interactions between NEST and multiprocessing.

Would it be best to add a note on this no-no to the Guide to parallel computing? Or do we also need a conceptual document? Could we make use of @juanmav's sample code?

sarakonradi commented 4 years ago

@suku248 Do you happen to know of any already existing conceptual document, where this has been looked into? Input from the kernel team is much appreciated.

suku248 commented 4 years ago

@sarakonradi I am not aware of any documentation on this topic. I don't think that the publications on PyNEST explicitly discourage users from using Python packages for multiprocessing on top of NEST's internal parallelization based on MPI/OpenMP.

I guess that such optimization attempts by NEST users typically target the network-construction phase (and not the actual simulation phase). Therefore, in the documentation for this issue it might be a good a idea to mention the work of Ippen et al. (2017), which shows that NEST is already good at building a network in parallel.