search

BlueBrain / Monsteer

Interactive Supercomputing Tools and Library
http://bluebrain.github.io/
GNU Lesser General Public License v3.0
4 stars 7 forks source link

Segmentation fault on importing Nesteer #7

Closed MartyMcFlyHasNoFarmHiaHiaHo closed 8 years ago

MartyMcFlyHasNoFarmHiaHiaHo commented 9 years ago

A segmentation fault is occuring when importing monsteer.Nesteer in python, for example with the provided steering example.

Computer specs: 4 cores, 16 GB ram Ubuntu 14 64 bit With python2.7 running in a virtual environment (mpi4py, scipy, numpy, etc.)

I am not sure if Monsteer causes the actual problem but maybe somebody got the same issue and knows a solution. In contrast to the minimal steering example, the non-steering example works fine along with the monsteer_spike_receiver app (which indicates that MUSIC works in general). Moreover, I tried both, the EPFL MUSIC repository and the official MUSIC repository. For both the same segmentation fault occurs. With nest 2.8 this segmentation fault did not occur (!) but python API mismatches due to changes in Nest.

If I import nest first and then monsteer.Nesteer, the segmentation fault does not occur, however other issues occur: " python: ../nestkernel/scheduler.h:736: librandom::RngPtr nest::Scheduler::get_rng(nest::thread) const: Assertion `thrd < staticcast(rng.size())' failed." (probably because music must set up the mpi environment at first)

Here is the traceback:

(neuro_64)schulze@....blabla.... $ mpirun -np 2 which music nest2music_proxy_with_steering.music [lada:30199] * Process received signal * [lada:30199] Signal: Segmentation fault (11) [lada:30199] Signal code: Address not mapped (1) [lada:30199] Failing at address: 0x38 [lada:30199] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f1120113340] [lada:30199] [ 1] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(+0x43d7ab) [0x7f11016e77ab] [lada:30199] [ 2] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(+0x43f223) [0x7f11016e9223] [lada:30199] [ 3] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(lt_dlopenadvise+0x52) [0x7f11016e9bf2] [lada:30199] [ 4] /usr/lib/x86_64-linux-gnu/hwloc/hwloc_xml_libxml.so(+0x13b3) [0x7f10fd3da3b3] [lada:30199] [ 5] /usr/lib/x86_64-linux-gnu/hwloc/hwloc_xml_libxml.so(+0x1ec3) [0x7f10fd3daec3] [lada:30199] [ 6] /usr/lib/x86_64-linux-gnu/libhwloc.so.5(+0x22056) [0x7f111b25b056] [lada:30199] [ 7] /usr/lib/x86_64-linux-gnu/libhwloc.so.5(+0x13e9f) [0x7f111b24ce9f] [lada:30199] [ 8] /usr/lib/x86_64-linux-gnu/libhwloc.so.5(hwloc_topology_set_xmlbuffer+0x49) [0x7f111b245858] [lada:30199] [ 9] /usr/lib/libmpi.so.1(opal_hwloc_unpack+0x8e) [0x7f111ad873fe] [lada:30199] [10] /usr/lib/libmpi.so.1(opal_dss_unpack_buffer+0x5e) [0x7f111ad82dbe] [lada:30199] [11] /usr/lib/libmpi.so.1(opal_dss_unpack+0x9a) [0x7f111ad83caa] [lada:30199] [12] /usr/lib/libmpi.so.1(orte_util_nidmap_init+0x291) [0x7f111ad4fef1] [lada:30199] [13] /usr/lib/openmpi/lib/openmpi/mca_ess_env.so(+0x1a49) [0x7f10ff901a49] [lada:30199] [14] /usr/lib/libmpi.so.1(orte_init+0x193) [0x7f111ad38843] [lada:30199] [15] /usr/lib/libmpi.so.1(ompi_mpi_init+0x174) [0x7f111acf8024] [lada:30199] [16] /usr/lib/libmpi.so.1(PMPI_Init_thread+0x16f) [0x7f111ad0f88f] [lada:30199] [17] /fzi/ids/schulze/local_setups/music_EPFL_64/lib/libmusic.so.100(_ZN3MPI11Init_threadERiRPPci+0x2a) [0x7f1100760ce6] [lada:30199] [18] /fzi/ids/schulze/local_setups/music_EPFL_64/lib/libmusic.so.100(_ZN5MUSIC5SetupC2ERiRPPciPi+0x102) [0x7f110075f932] [lada:30199] [19] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN4nest12Communicator4initEPiPPPc+0x145) [0x7f1101688d75] [lada:30199] [20] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(_Z11neststartupiPPcR14SLIInterpreterRPN4nest7NetworkESs+0x31) [0x7f11014fd899] [lada:30199] [21] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(+0x218f52) [0x7f11014c2f52] [lada:30199] [22] /fzi/ids/schulze/local_setups/nest-2.6.0_64/lib/python2.7/site-packages/nest/pynestkernel.so(+0x21841d) [0x7f11014c241d] [lada:30199] [23] python(PyEval_EvalFrameEx+0x40d) [0x49968d] [lada:30199] [24] python(PyEval_EvalFrameEx+0xc72) [0x499ef2] [lada:30199] [25] python(PyEval_EvalCodeEx+0x2ac) [0x4a090c] [lada:30199] [26] python(PyEval_EvalCode+0x32) [0x588d42] [lada:30199] [27] python(PyImport_ExecCodeModuleEx+0xaa) [0x588dfa] [lada:30199] [28] python() [0x5b1f0f] [lada:30199] [29] python() [0x44d4d2]

[lada:30199] * End of error message *

mpirun noticed that process rank 0 with PID 30199 on node lada exited on signal 11 (Segmentation fault).

jafyvilla commented 9 years ago

@MartyMcFlyHasNoFarmHiaHiaHo I have never seen that specific error, but I have experienced some problems related to the order of the imports. From our experience, NEST should be imported in Python before Nesteer, otherwise some libraries run into some conflicts, apparently. You should be safe doing it in that order: import nest and then import Nesteer. Also, using recent versions of NEST we have seen other problems (different than yours), but 2.4.2 works fine for us, if you want to give it a try.

You might want to check with the official NEST repository or mailing list too, maybe they can help you if the error is coming from NEST.

MartyMcFlyHasNoFarmHiaHiaHo commented 9 years ago

Nest 2.4.2 didnt worked for me either but I tried both (2.6.0 and 2.4.2) on 32 bits and it works ... nearly. Instead of the segmentation fault above there is now another error (I think the output of 2.6.0 is more helpful). Ill try to track that problem now but = again = maybe somebody knows the issue already

With nest 2.6.0: (nest2.6)schulze@bentley:/disk/no_backup/schulze/builds_32/Monsteer/share/Monsteer/examples/nest$ mpirun -np 2 which music nest2music_proxy_with_steering.music PID.Thread | Filename:line | ms | Message 7372.Unknown ox/lunchbox/pluginFactory.ipp:155 1225 Enabled plugin libBrionMonsteerSpikeReport.so Bound to monsteer://bentley:56484 7372.Unknown Monsteer/apps/music_proxy.cpp:50 1237 Initializing Spikes Handler 7372.Unknown Monsteer/apps/music_proxy.cpp:77 1237 Initialized Spikes Handler 7372.Unknown Monsteer/apps/music_proxy.cpp:193 1240 Initializing Steering Handler 7372.Unknown Monsteer/apps/music_proxy.cpp:213 1240 Initialized Steering Handler 7372.Unknown Monsteer/apps/music_proxy.cpp:300 1240 Starting MUSIC runtime

          -- N E S T --

Copyright (C) 2004 The NEST Initiative Version 2.6.0 Oct 26 2015 16:58:52

This program is provided AS IS and comes with NO WARRANTY. See the file LICENSE for details.

Problems or suggestions? Website : http://www.nest-initiative.org Mailing list: nest_user@nest-initiative.org

Type 'nest.help()' to find out more about NEST.

Oct 26 17:04:30 Simulate [Info]: Simulating 10 ms.

Oct 26 17:04:30 Scheduler::prepare_nodes [Info]: Please wait. Preparing elements. Traceback (most recent call last): File "./minimal_steered.py", line 34, in Simulate(10) File "/disk/no_backup/schulze/builds_32/nest-2.6.0/lib/python2.7/site-packages/nest/hl_api.py", line 153, in stack_checker_func return f(_args, *_kwargs) File "/disk/no_backup/schulze/builds_32/nest-2.6.0/lib/python2.7/site-packages/nest/hl_api.py", line 361, in Simulate sr('ms Simulate') File "/disk/no_backup/schulze/builds_32/nest-2.6.0/lib/python2.7/site-packages/nest/init.py", line 81, in catching_sli_run raise hl_api.NESTError("{0} in {1}{2}".format(errorname, commandname, message)) pynestkernel.NESTError: UnknownReceptorType in Simulate_d: Receptor type 10000000 is not available in music_event_out_proxy.

With nest 2.4.2:

(nest2.6)schulze@bentley:/disk/no_backup/schulze/builds_32/Monsteer/share/Monsteer/examples/nest$ mpirun -np 2 which music nest2music_proxy_with_steering.music PID.Thread | Filename:line | ms | Message 7425.Unknown ox/lunchbox/pluginFactory.ipp:155 1168 Enabled plugin libBrionMonsteerSpikeReport.so Bound to monsteer://bentley:49540 7425.Unknown Monsteer/apps/music_proxy.cpp:50 1177 Initializing Spikes Handler 7425.Unknown Monsteer/apps/music_proxy.cpp:77 1177 Initialized Spikes Handler 7425.Unknown Monsteer/apps/music_proxy.cpp:193 1187 Initializing Steering Handler 7425.Unknown Monsteer/apps/music_proxy.cpp:213 1187 Initialized Steering Handler 7425.Unknown Monsteer/apps/music_proxy.cpp:300 1188 Starting MUSIC runtime

          -- N E S T --

Copyright (C) 2004 The NEST Initiative Version 2.4.2 Oct 26 2015 16:53:55

This program is provided AS IS and comes with NO WARRANTY. See the file LICENSE for details.

Problems or suggestions? Website : http://www.nest-initiative.org Mailing list: nest_user@nest-initiative.org

Type 'nest.help()' to find out more about NEST.

Oct 26 17:04:41 Simulate [Info]: Simulating 10 ms.

Oct 26 17:04:41 Scheduler::prepare_nodes [Info]: Please wait. Preparing elements. terminate called after throwing an instance of 'nest::UnknownReceptorType' what(): UnknownReceptorType [bentley:07424] * Process received signal * [bentley:07424] Signal: Aborted (6) [bentley:07424] Signal code: (-6) [bentley:07424] [ 0] [0xb77de410] [bentley:07424] [ 1] [0xb77de428] [bentley:07424] [ 2] /lib/i386-linux-gnu/libc.so.6(gsignal+0x47) [0xb7604607] [bentley:07424] [ 3] /lib/i386-linux-gnu/libc.so.6(abort+0x143) [0xb7607a33] [bentley:07424] [ 4] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27verbose_terminate_handlerEv+0x195) [0xb6c36405] [bentley:07424] [ 5] /usr/lib/i386-linux-gnu/libstdc++.so.6(+0x4a063) [0xb6c34063] [bentley:07424] [ 6] /usr/lib/i386-linux-gnu/libstdc++.so.6(+0x48f95) [0xb6c32f95] [bentley:07424] [ 7] /usr/lib/i386-linux-gnu/libstdc++.so.6(gxx_personality_v0+0x519) [0xb6c33b99] [bentley:07424] [ 8] /lib/i386-linux-gnu/libgcc_s.so.1(+0x14f7b) [0xb6be1f7b] [bentley:07424] [ 9] /lib/i386-linux-gnu/libgcc_s.so.1(_Unwind_RaiseException+0x11e) [0xb6be22ee] [bentley:07424] [10] /usr/lib/i386-linux-gnu/libstdc++.so.6(__cxa_throw+0x69) [0xb6c342f9] [bentley:07424] [11] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN4nest21music_event_out_proxy9calibrateEv+0xe9f) [0xb306254f] [bentley:07424] [12] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(+0x353e08) [0xb312ee08] [bentley:07424] [13] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN4nest9Scheduler13prepare_nodesEv+0xa7) [0xb3132097] [bentley:07424] [14] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN4nest9Scheduler18prepare_simulationEv+0x117) [0xb3132dc7] [bentley:07424] [15] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN4nest9Scheduler8simulateERKNS_4TimeE+0x1d0) [0xb3133250] [bentley:07424] [16] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN4nest7Network8simulateERKNS_4TimeE+0x26) [0xb3110cb6] [bentley:07424] [17] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZNK4nest10NestModule16SimulateFunction7executeEP14SLIInterpreter+0x2af) [0xb310624f] [bentley:07424] [18] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN14SLIInterpreter8execute_Ej+0x255) [0xb31721f5] [bentley:07424] [19] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(_ZN14SLIInterpreter7executeERKSs+0x191) [0xb3172821] [bentley:07424] [20] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(+0x175516) [0xb2f50516] [bentley:07424] [21] /disk/no_backup/schulze/builds_32/nest-2.4.2/lib/python2.7/site-packages/nest/pynestkernel.so(+0x1751fc) [0xb2f501fc] [bentley:07424] [22] python(PyEval_EvalFrameEx+0xca7) [0x8197397] [bentley:07424] [23] python(PyEval_EvalFrameEx+0x998) [0x8197088] [bentley:07424] [24] python() [0x80f8bc2] [bentley:07424] [25] python(PyEval_EvalFrameEx+0x252c) [0x8198c1c] [bentley:07424] [26] python(PyEval_EvalCodeEx+0x212) [0x81089a2] [bentley:07424] [27] python(PyEval_EvalFrameEx+0x79b) [0x8196e8b] [bentley:07424] [28] python(PyEval_EvalCode+0x122) [0x819dd92] [bentley:07424] [29] python() [0x81f01ed]

[bentley:07424] * End of error message *

mpirun noticed that process rank 0 with PID 7424 on node bentley exited on signal 6 (Aborted).

MartyMcFlyHasNoFarmHiaHiaHo commented 9 years ago

Update

The issue seems to be in the Nesteer.py whereas in the _setup_music method a dummy neuron is added @ music channel 10000000 and connected to a music output neuron.

In the nest::music_event_out_proxy::calibrate() function there is a check " if there are connections to receiver ports, which are beyond the width of the port", so I guess the example minimal_steered.py was just not adjusted to the new Nesteer.py. Ill check that tomorrow and let you know

jafyvilla commented 9 years ago

@MartyMcFlyHasNoFarmHiaHiaHo you are right, thanks for digging into it. It's not in minimal_steered.py though, but in the music configuration file (nest2music_proxy_with_steering.music), it should be:

from.spikes_out -> to.spikesPort [10000000]

Sorry, somehow we missed that example after changing the Nesteer class. Ideally that number should be the same as the highest cell id that you are trying to simulate, if I remember correctly, but for some of our use cases this number was varying, so having a number high enough in the configuration was the most straight away solution.

hernando commented 8 years ago

I'm closing this issue. Reopen if you think it needs any further action.