Open RCagnol opened 1 year ago
probably related to #657
It seems that the issue is caused by the fact that only some of the MPI processes try to access to self._simulator.state.t. Indeed, it seems that Nest freezes when only some MPI nodes access to nest.biological_time.
For example, the following nest code freezes when ran with mpirun -n 2
from mpi4py import MPI
mpi_comm = MPI.COMM_WORLD
import nest
if mpi_comm.rank == 0:
nest_time = nest.biological_time
print(nest_time)
I noticed that in PyNN 0.10.1 the simulations freeze when the number of MPI node exceedes the number of neurons in a population. This wasn't happening in PyNN 0.10.0.
For example the following code freezes when the number of MPI process is higher than 2:
The MPI nodes to which no neurons are assigned seems to reach the call to gather here: https://github.com/NeuralEnsemble/PyNN/blob/master/pyNN/recording/__init__.py#L73
While the MPI nodes to which some neurons are assigned seem to freeze before that, when trying to access to
self._simulator.state.t
in different parts of the _get_current_segment() method ( https://github.com/NeuralEnsemble/PyNN/blob/master/pyNN/recording/__init__.py#L269 )I didn't manage to figure out what is exactly causing that, but it seems that in the same method, displacing the following lines:
before the
if signal_array.size > 0:
condition (which was the case in PyNN 0.10.0) seems somehow to solve the issue.