Cython error when calling brian2 from nested loops ('Import Error: (...) cannot change memory protections')

[wilhelmbraun]

Dear all,

I have recently encountered an issue with cython and/ or brian2 when calling a custom function executing brian2 code in multiple nested loops. I am not sure whether it is a problem with cython or brian2 (especially with the PoissonInput class), or whether it is simply a memory issue. The offending code essentially looks as follows (sorry for missing indentations):

from brian2 import * prefs.codegen.target = 'cython'

def custom_function(some_parameters): start_scope() (brian2 code setting up neurons, networks and synapses, all with 'magic' syntax) run(some_duration) return some_result

for x in param1_array: for y in param2_array: result = custom_function(x,y)

Now, the error message shown below after the dashed line appears after several 100 calls of 'custom_function' (apologies for posting it here in all its length). Any hints on how to avoid this (probably trivial) problem would be very much appreciated.

Thanks a lot.

Best, Wilhelm

---

ERROR Brian 2 encountered an unexpected error. If you think this is bug in Brian 2, please report this issue either to the mailing list at http://groups.google.com/group/brian-development/, or to the issue tracker at https://github.com/brian-team/brian2/issues. Please include this file with debug information in your report: /tmp/brian_debug_87il1crn.log Additionally, you can also include a copy of the script that was run, available at: /tmp/brian_script_ulrzdcyt.py You can also include a copy of the redirected std stream outputs, available at /tmp/brian_stdout0mp407k.log and /tmp/brian_stderr_js8w77dq.log Thanks! [brian2] Traceback (most recent call last): File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/network.py", line 841, in before_run obj.before_run(run_namespace) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/input/poissoninput.py", line 109, in before_run CodeRunner.before_run(self, run_namespace=run_namespace) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/groups/group.py", line 1128, in before_run codeobj_class=self.codeobj_class File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/codeobject.py", line 416, in create_runner_codeobj compiler_kwds=compiler_kwds File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/devices/device.py", line 326, in code_object codeobj.compile() File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/runtime/cython_rt/cython_rt.py", line 147, in compile sources=self.sources File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/runtime/cython_rt/extension_manager.py", line 138, in create_extension sources=sources) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/runtime/cython_rt/extension_manager.py", line 299, in _load_module module = imp.load_dynamic(module_name, module_path) File "/home/wilhelm/brian2/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) File "", line 684, in _load File "", line 658, in _load_unlocked File "", line 571, in module_from_spec File "", line 922, in create_module File "", line 219, in _call_with_frames_removed ImportError: /home/wilhelm/.cython/brian_extensions/_cython_magic_45df6409df9ce801f9cf1b2537955e60.cpython-36m-x86_64-linux-gnu.so: cannot change memory protections

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "script.py", line 562, in result = custom_function(...) File "script.py", line 342, in custom_function run(some_duration) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/units/fundamentalunits.py", line 2375, in new_f result = f(*args, kwds) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/magic.py", line 371, in run namespace=namespace, profile=profile, level=2+level) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/magic.py", line 231, in run namespace=namespace, profile=profile, level=level+1) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/base.py", line 276, in device_override_decorated_function return func(*args, *kwds) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/units/fundamentalunits.py", line 2375, in new_f result = f(args, kwds) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/network.py", line 951, in run self.before_run(namespace) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/base.py", line 276, in device_override_decorated_function return func(*args, **kwds) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/network.py", line 843, in before_run raise brian_object_exception("An error occurred when preparing an object.", obj, ex) brian2.core.base.BrianObjectException: Original error and traceback: Traceback (most recent call last): File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/core/network.py", line 841, in before_run obj.before_run(run_namespace) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/input/poissoninput.py", line 109, in before_run CodeRunner.before_run(self, run_namespace=run_namespace) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/groups/group.py", line 1128, in before_run codeobj_class=self.codeobj_class File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/codeobject.py", line 416, in create_runner_codeobj compiler_kwds=compiler_kwds File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/devices/device.py", line 326, in code_object codeobj.compile() File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/runtime/cython_rt/cython_rt.py", line 147, in compile sources=self.sources File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/runtime/cython_rt/extension_manager.py", line 138, in create_extension sources=sources) File "/home/wilhelm/brian2/lib/python3.6/site-packages/brian2/codegen/runtime/cython_rt/extension_manager.py", line 299, in _load_module module = imp.load_dynamic(module_name, module_path) File "/home/wilhelm/brian2/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) File "", line 684, in _load File "", line 658, in _load_unlocked File "", line 571, in module_from_spec File "", line 922, in create_module File "", line 219, in _call_with_frames_removed ImportError: /home/wilhelm/.cython/brian_extensions/_cython_magic_45df6409df9ce801f9cf1b2537955e60.cpython-36m-x86_64-linux-gnu.so: cannot change memory protections

Error encountered with object named "poissoninput". Object was created here (most recent call only, full details in debug log): File "script.py", line 336, in custom_function Excitatory_Poisson_Group = PoissonInput(...)

An error occurred when preparing an object. ImportError: /home/wilhelm/.cython/brian_extensions/_cython_magic_45df6409df9ce801f9cf1b2537955e60.cpython-36m-x86_64-linux-gnu.so: cannot change memory protections (See above for original error message and traceback.)

[mstimberg]

Thanks for the report, I've never seen this error before... Could it have something to do with SELINUX or other protection mechanisms on Linux? Can you reproduce the problem with a very simple network (say one NeuronGroup)?

Do you really create a new network for each iteration? What is the difference between the networks? Apart from this error, it would be much more efficient to use store()/restore() with a single network if all you need to do is change parameters, etc.

[wilhelmbraun]

Thanks for your quick reply. I have not yet checked whether the issue can be reproduced with a very simple network. Yes, I create a new network for every iteration of a given parameter set. The first reason is that I want to average over different connectivities and noise realizations for fixed parameters (in a Monte-Carlo like fashion). The second reason is that I also want to change noise realizations and connectivities when I choose a new parameter set, so that everything is properly randomized. I understand that usingstore()/ restore() would help in reducing the overhead from creating new network objects, at the expense that e.g. the connectivity matrix cannot be re-drawn for every new run. Is there a way to re-draw connectivities and noise realizations afresh without having to re-create an entire new network, and if so , would it reduce overhead? Thanks!

[mstimberg]

With "noise realizations" do you mean the noise in stochastic equations using xi, and/or the use of rand() or randn() in expressions used during a run? Both of these would always be fresh realizations, even if you use store/restore (to have them fixed you'll have to use seed(...)).

For random connections and random parameter initializations, you can still combine this with store/restore if you create the connections/initialize the parameters after the restore. E.g. the following will not build the network from scratch every time, but still have different random connections and different initializations for each run:

# Do everything that is the same across simulations
group1 = NeuronGroup(...)
group2 = NeuronGroup(...)
synapses = Synapses(group1, group2, ...)
store()
for ... in ...:
   restore()
   # Now initialize everything that should vary across repetitions
   synapses.connect(...)
   group1.v = '-70*mV+rand()*10*mV'
   ...

[wilhelmbraun]

Thanks for this super helpful answer, I will try it out and see whether the error persists and if so, try to reproduce it with a minimal example.

[wilhelmbraun]

So I implemented my networks using store/restore, and the error does not appear any more. However, I am not sure about the use of SpikeMonitor or PopulationRateMonitor with this feature. In particular, in the for loops, I want to change both parameters of some PoissonInput (e.g. their rate) and add a new PoissonGroup, along with synapses, as well as reinstating synaptic connections using synapses.connect().

I set up my network using

net = Network(neuron_groups, synapses_1, monitors)  
net.store()  
#change both parameters (e.g. for a PoissonGroup or synaptic connection probability) 
#and realizations of synaptic connectivity in this for loop
for ... in ...
 net.restore()
 #reinstate random connections, e.g.
 synapses_1.connect(...)
 #Add PoissonGroups (not with net.add(), but by just writing 
 poisson_group_1 = PoissonInput(neuron_group_1,...)
 #and set up parameters of PoissonInput.
 #Also add a PoissonGroup and a new synapse object for this group:
 driving_group = PoissonGroup(...)
 syn_external = Synapses(driving_group, neuron_group_1, ...)
 syn_external.connect(p=...)
 #It's important that all inputs from PoissonInput and PoissonGroup classes are changed during every run.
 net.run(some_duration)
 #use data from the earlier added Spike and PopulationRateMonitor to compute network characteristics

Now, the problem is that I get a lot of non-sensical output from the quantities computed using PopulationRateMonitor, so maybe they should be set up in the for loop? Or maybe brian finds it confusing that neuron groups setup using net later receive input in the for loop that is not part of the net? I did read this tutorial: https://brian2.readthedocs.io/en/stable/resources/tutorials/3-intro-to-brian-simulations.html, but could not find an easy fix. Thanks a lot for your help!

[wilhelmbraun]

I think I found the error. The issue is indeed that in the for loop, after calling net.restore(), poisson_group_1 and driving_group are not added to the net and hence not simulated. Adding them after net.restore() was not possible. Using magic syntax resulted in a MagicError. Is there a way to get around this? Thanks again!

[mstimberg]

Hi. All your objects that you want to simulate have indeed to be part of the Network object. However, you should be able to add them later. What do you mean with: "Adding them after net.restore() was not possible."? The only thing you have to be careful with, is that at the point of net.restore(), the network has to consist of the same objects as when you called net.store(), i.e. if you use net.add after net.store(), then you'll have to use net.remove before you call net.restore().

And just to make sure: you want to change the parameters of the PoissonGroup/PoissonInput between runs, not just have new realizations of the random processes, correct? As I said before, the random realizations will be independent across runs, even without reconstructing the objects.

[wilhelmbraun]

Hi again, thanks- I indeed forgot to add net.remove() after net.add(). Now, the code runs as it should.

And just to make sure: you want to change the parameters of the PoissonGroup/PoissonInput between runs, not just have new realizations of the random processes, correct? As I said before, the random realizations will be independent across runs, even without reconstructing the objects.

Yes, I want to change parameters and realizations of the random process, which now can be achieved using store()/restore(). I understand from your comment above that this would not be necessary with PoissonGroup/ PoissonInput, because even if they are added before net.store(), they would be re-drawn when net.run() is called?

[mstimberg]

I understand from your comment above that this would not be necessary with PoissonGroup/ PoissonInput, because even if they are added before net.store(), they would be re-drawn when net.run() is called?

Random numbers are drawn continuously, independent of store/restore, they are only set to a specific starting point if you use the seed function. Therefore, if you only need new realizations of random numbers, then you can add them in the beginning before net.store(). However, if you want to change their parameters (e.g. their rates), then you have to recreate them, since they currently do not allow for changes in parameters after creation.

[mstimberg]

Closing this for now since we never reproduced the original error and some other issues haven been successfully resolved. Feel free to open a new issue (or reopen this one) if the Cython error comes up again.