search

jgarridoalcazar / SpikingCerebellum

2 stars 1 forks source link

Creation of synapses of type stdp_cos fails: "terminate called without an active exception" #2

Closed lguyot closed 6 years ago

lguyot commented 6 years ago

As explained in https://hbpneurorobotics.atlassian.net/browse/NUIT-36, If I run the following code snippet

import pyNN.nest as sim
import numpy as np
mf_vn_weights = 0.001 
num_MF_neurons = 1
num_VN_neurons = 2
sim.setup(timestep=0.5, threads=4, min_delay=0.5, max_delay=100.0)
def create_brain():
    VN_PARAMS = {'C_m': 0.002,
                 'g_L': 0.0002, 
                 'E_L': -70.0,
                 'E_ex': 0.0,
                 'E_in': -80.0,
                 'e_ts': 0.0,
                 'V_reset': -70.5,
                 'V_th': -40.0,
                 't_ref': 1.0,
                 'tau_syn_ex': 0.5,
                 'tau_syn_in': 7.0,
                 'tau_syn_ts': 0.85,
                 'tau_cos': 10.0,
                 'exponent': 2.0}

    parrot_neuron = sim.native_cell_type('parrot_neuron')
    MF_population = sim.Population(num_MF_neurons,parrot_neuron(),label='MFLayer')
    vn_neuron = sim.native_cell_type('iaf_cond_exp_cos')
    VN_population = sim.Population(num_VN_neurons,vn_neuron(**VN_PARAMS),label='VNLayer')
    stdp_cos = sim.native_synapse_type('stdp_cos_synapse')(**{'weight':mf_vn_weights,
                                                              'delay':1.0,
                                                              'exponent': 2.0,
                                                              'tau_cos': 5.0,
                                                              'A_plus': 0.0000009,
                                                              'A_minus': 0.00001,
                                                              'Wmin': 0.0005,
                                                              'Wmax': 0.007})

    mf_vn_connections = sim.Projection(MF_population,
                    VN_population,
                    sim.AllToAllConnector(),
                    receptor_type='AMPA',
                    synapse_type = stdp_cos)

    return MF_population
circuit = create_brain()

I get

lguyot@bluebrain146:~$ source ~/.opt/platform_venv/bin/activate
(platform_venv) lguyot@bluebrain146:~$ python iCub_VOR_UGR_brain_GOLGI.py 
[INFO] [2018.8.6 15:1:35 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.8.6 15:1:35 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:221 @ Network::create_rngs_] : Deleting existing random number generators
[INFO] [2018.8.6 15:1:35 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.8.6 15:1:35 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:272 @ Network::create_grng_] : Creating new default global RNG

              -- N E S T --

  Copyright (C) 2004 The NEST Initiative
  Version 2.12.0 Aug  6 2018 10:33:52

This program is provided AS IS and comes with
NO WARRANTY. See the file LICENSE for details.

Problems or suggestions?
  Visit http://www.nest-simulator.org

Type 'nest.help()' to find out more about NEST.
CSAConnector: libneurosim support not available in NEST.
Falling back on PyNN's default CSAConnector.
Please re-compile NEST using --with-libneurosim=PATH
terminate called without an active exception
Aborted

The above error is related to C++ multithreading, see https://stackoverflow.com/questions/7381757/c-terminate-called-without-an-active-exception.

jgarridoalcazar commented 6 years ago

I don't think that this issue is directly related to multithreading. My guess is that any of the threads is using memory out of the allocated space and terminating with segmentation fault, so that the thread manager is throwing the exception.

Can you change line 6 of your code to the following?: sim.setup(timestep=0.5, threads=1, min_delay=0.5, max_delay=100.0)

I have tested with several number of threads and the code always finishes correctly. Hopefully, changing the number of threads will unveil the real issue.

jgarridoalcazar commented 6 years ago

@lguyot, could you also run the script src/testSTDPCos_New.py and let me know if it shows the ending plots? This script uses the same synapse type, but calling it directly from PyNEST.

lguyot commented 6 years ago

@jgarridoalcazar Dear Jesus, I set the number of threads to 1 and the result remains the same.

Here is what happens when I run testSTDPCos_New.py:

`(platform_venv) ✔ ~/SpikingCerebellum/src [master|✔] 09:36 $ python testSTDPCos_New.py [INFO] [2018.8.15 9:37:1 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::createrngs] : Creating default RNGs [INFO] [2018.8.15 9:37:1 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:221 @ Network::createrngs] : Deleting existing random number generators [INFO] [2018.8.15 9:37:1 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::createrngs] : Creating default RNGs [INFO] [2018.8.15 9:37:1 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:272 @ Network::creategrng] : Creating new default global RNG

          -- N E S T --

Copyright (C) 2004 The NEST Initiative Version 2.12.0 Aug 6 2018 17:07:17

This program is provided AS IS and comes with NO WARRANTY. See the file LICENSE for details.

Problems or suggestions? Visit http://www.nest-simulator.org

Type 'nest.help()' to find out more about NEST. Segmentation fault `

jgarridoalcazar commented 6 years ago

Thanks @lguyot. Can you isolate the line in testSTDPCos_New.py where the segmentation fault occurs?

lguyot commented 6 years ago

Actually, I just needed to remove the nest.Install('cerebellummodule') cerebellum command. Afterwards, there is a problem with "pylab.figure()" (AttributeError: 'figure()' is not a Qt property or a signal), which is not relevant to us. Once I removed all the matplotlib instructions, I can see that the network is built and simulated successfully:

(platform_venv) ✘-1 ~/SpikingCerebellum/src [master|✚ 1] 
11:04 $ python testSTDPCos_New.py
[INFO] [2018.8.15 11:4:40 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.8.15 11:4:40 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:221 @ Network::create_rngs_] : Deleting existing random number generators
[INFO] [2018.8.15 11:4:40 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.8.15 11:4:40 /home/lguyot/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:272 @ Network::create_grng_] : Creating new default global RNG

              -- N E S T --

  Copyright (C) 2004 The NEST Initiative
  Version 2.12.0 Aug  6 2018 17:07:17

This program is provided AS IS and comes with
NO WARRANTY. See the file LICENSE for details.

Problems or suggestions?
  Visit http://www.nest-simulator.org

Type 'nest.help()' to find out more about NEST.

Aug 15 11:04:41 ModelManager::clear_models_ [Info]: 
    Models will be cleared and parameters reset.

Aug 15 11:04:41 Network::create_rngs_ [Info]: 
    Deleting existing random number generators

Aug 15 11:04:41 Network::create_rngs_ [Info]: 
    Creating default RNGs

Aug 15 11:04:41 Network::create_grng_ [Info]: 
    Creating new default global RNG

Aug 15 11:04:41 NodeManager::prepare_nodes [Info]: 
    Preparing 2005 nodes for simulation.

Aug 15 11:04:41 MUSICManager::enter_runtime [Info]: 
    Entering MUSIC runtime with tick = 1 ms

Aug 15 11:04:41 SimulationManager::resume [Info]: 
    Number of local nodes: 2005
    Simulaton time (ms): 1550
    Number of OpenMP threads: 1
    Number of MPI processes: 1

Aug 15 11:04:41 SimulationManager::resume [Info]: 
    Simulation finished.

So it seems that the problem lies on the PyNN side.

jgarridoalcazar commented 6 years ago

@lguyot That could be an explanation. The other one, some memory issue in the NEST module code that does not always shows. Still trying to reproduce the issue. Which PyNN version are you using in your environment? PyNN 0.9.2 here.

lguyot commented 6 years ago

@jgarridoalcazar I am using PyNN==0.9.0 (this is the version installed by with NRP, se CLE/hbp_nrp_cle/requirements.txt).

Upgrading to PyNN==0.9.2 fixes the issue, in the sense that python iCub_VOR_UGR_brain_GOLGI.py runs without the final error message terminate called without an active exception Aborted

jgarridoalcazar commented 6 years ago

That is a good piece of news. Checking the PyNN repository there exist many changes in the code of the projection setting for NEST since PyNN 0.9.0. My desktop computer has 0.9.1 and it seems to work too. We have to check the PyNN version in the computer where the NRP is working (and the VOR example has been developed).

@FranciscoNaveros can you check the PyNN version the desktop computer? In a python terminal run import pyNN and pyNN.version

jgarridoalcazar commented 6 years ago

We close this issue as it seems to be fixed using PyNN 0.9.1 and higher