search

neuronsimulator / nrn

NEURON Simulator
http://nrn.readthedocs.io
Other
378 stars 114 forks source link

segfault when running the tests #2867

Open mgeplf opened 2 months ago

mgeplf commented 2 months ago

Context

Overview of the issue

I saw the pytest pin, and was wondering if I could get it to happen with the older version of pytest

Expected result/behavior

Not have a segfault

NEURON setup

Commit 8a9bada6; python 3.11 from pyenv on Ubuntu 20.04.

(fresh virtualenv)
$ cmake -DNRN_ENABLE_CORENEURON=ON -DNRN_ENABLE_PYTHON=ON -DNRN_ENABLE_TESTS=ON ..
$ pip install -r ../nrn_requirements.txt
$ pip install -r ../external/nmodl/requirements.txt
$ ninja
$ ctest --verbose -R pytest_coreneuron::basic_tests_py3.11
# use the output of the command to rerun tests:
$ cmake "-E" "env" [...] "python3" "-m" "pytest" "--capture=tee-sys" "./test/pytest_coreneuron"

If I run it a few times, along with failures, I get a SEGFAULT.

In gdb, I get the following traceback:

test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_swc.py test 0
error temp.tmp line 1: could not parse:

# One-point soma
#n,type,x,y,z,radius,parent
1  1 0 0 0 10 -1
2  3 0 10 0 1 1
3  3 0 20 0 1 2
4  3 0 -10 0 2 1
5  3 0 -20 0 2 4
6  3 10 0 0 .5 1
7  3 10 5 0 1 6

Thread 1 received signal SIGSEGV, Segmentation fault.
neuron::container::handle_base<neuron::container::owning_identifier<neuron::container::Node::storage> >::get_handle<neuron::container::Node::field::Voltage> (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/neuron/container/view_utils.hpp:93
93              return underlying_storage().template get_handle<Tag>(this->id());
(gdb) bt
#0  neuron::container::handle_base<neuron::container::owning_identifier<neuron::container::Node::storage> >::get_handle<neuron::container::Node::field::Voltage> (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/neuron/container/view_utils.hpp:93
#1  neuron::container::Node::handle_interface<neuron::container::owning_identifier<neuron::container::Node::storage> >::v_handle (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/neuron/container/node.hpp:187
#2  Node::v_handle (this=0x55d9da52b2d0) at /home/gevaert/tmp/nrn/src/nrnoc/section.h:153
#3  mk_ttd () at /home/gevaert/tmp/nrn/src/nrniv/partrans.cpp:511
#4  0x00007f47a4780d27 in nrn_thread_memblist_setup () at /home/gevaert/tmp/nrn/src/nrnoc/multicore.cpp:647
#5  0x00007f47a47776dd in v_setup_vectors () at /home/gevaert/tmp/nrn/src/nrnoc/treeset.cpp:1697
#6  0x00007f47a476e913 in nrnhoc_topology () at /home/gevaert/tmp/nrn/src/nrnoc/solve.cpp:296
#7  0x00007f47a478a0e7 in hoc_call () at /home/gevaert/tmp/nrn/src/oc/code.cpp:1418
#8  0x00007f47a4816f1a in fcall (vself=vself@entry=0x7f47a4ccdb90, vargs=vargs@entry=0x7f47a6d8bb18 <_PyRuntime+58904>) at /home/gevaert/tmp/nrn/src/nrnpython/nrnpy_hoc.cpp:728
#9  0x00007f47a4710ace in OcJump::fpycall (f=f@entry=0x7f47a4816dd0 <fcall(void*, void*)>, a=a@entry=0x7f47a4ccdb90, b=b@entry=0x7f47a6d8bb18 <_PyRuntime+58904>) at /home/gevaert/tmp/nrn/src/nrniv/ocjump.cpp:138
#10 0x00007f47a4813f0a in hocobj_call (self=0x7f47a4ccdb90, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwrds=<optimized out>) at /home/gevaert/tmp/nrn/src/nrnpython/nrnpy_hoc.cpp:796
#11 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a4ccdb90, args=<optimized out>, nargs=0, keywords=0x0) at Objects/call.c:214
#12 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#13 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df86c0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#14 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#15 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x7f479d5b6640, tuple=0x7f47a6d8bb18 <_PyRuntime+58904>, callable=0x7f479ec9ade0, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#16 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f479ec9ade0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=0x7f479d5b6640) at Objects/call.c:328
#17 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x7f479d5b6640, callargs=0x7f47a6d8bb18 <_PyRuntime+58904>, func=0x7f479ec9ade0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#19 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df8608, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#20 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#21 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f479f865e70, callable=0x7f47a58b3560, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#22 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a58b3560, args=0x7f479f865e70, kwargs=0x0) at Objects/call.c:328
#23 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f479f865e70, func=0x7f47a58b3560, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#24 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#25 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df83a0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#26 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#27 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641abd0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f479d5b69c0) at Objects/call.c:152
#28 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b510, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f479d5b69c0) at Objects/call.c:482
#29 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b510, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f479d5b69c0) at Objects/typeobject.c:7630
#30 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b510, args=<optimized out>, nargs=0, keywords=0x7f47a58bdc00) at Objects/call.c:214
#31 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#32 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df82c0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#33 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#34 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f479e434040, callable=0x7f47a59dca40, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#35 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dca40, args=0x7f479e434040, kwargs=0x0) at Objects/call.c:328
#36 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f479e434040, func=0x7f47a59dca40, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#37 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#38 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df8058, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#39 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#40 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641b0c0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f479ecb46c0) at Objects/call.c:152
#41 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b6f0, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f479ecb46c0) at Objects/call.c:482
#42 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b6f0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f479ecb46c0) at Objects/typeobject.c:7630
#43 0x00007f47a6969728 in _PyObject_Call (kwargs=0x7f479ecb46c0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, callable=0x7f47a576b6f0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:343
#44 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b6f0, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=0x7f479ecb46c0) at Objects/call.c:313
#45 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x7f479ecb46c0, callargs=0x7f47a6d8bb18 <_PyRuntime+58904>, func=0x7f47a576b6f0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#46 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#47 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7cf8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#48 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#49 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f479e3b6900, callable=0x7f47a59dc7c0, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#50 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dc7c0, args=0x7f479e3b6900, kwargs=0x0) at Objects/call.c:328
#51 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f479e3b6900, func=0x7f47a59dc7c0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#52 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#53 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7a90, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#54 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#55 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641b5c0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f479eca3640) at Objects/call.c:152
#56 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b880, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f479eca3640) at Objects/call.c:482
#57 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b880, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f479eca3640) at Objects/typeobject.c:7630
#58 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b880, args=<optimized out>, nargs=0, keywords=0x7f47a5950300) at Objects/call.c:214
#59 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#60 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df79f8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
[4:15](https://bluebrainproject.slack.com/archives/C071MBESA6R/p1714659301772289)
#61 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#62 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f47a4cda170, callable=0x7f47a59dd8a0, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#63 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dd8a0, args=0x7f47a4cda170, kwargs=0x0) at Objects/call.c:328
#64 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f47a4cda170, func=0x7f47a59dd8a0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#65 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#66 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7790, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#67 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#68 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641bab0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f47a4cbda00) at Objects/call.c:152
#69 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576b970, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f47a4cbda00) at Objects/call.c:482
#70 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576b970, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f47a4cbda00) at Objects/typeobject.c:7630
#71 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576b970, args=<optimized out>, nargs=0, keywords=0x7f47a5f4f700) at Objects/call.c:214
#72 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#73 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df75e8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#74 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#75 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f47a4fa5ea0, callable=0x7f47a59dd760, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#76 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a59dd760, args=0x7f47a4fa5ea0, kwargs=0x0) at Objects/call.c:328
#77 0x00007f47a6908618 in do_call_core (use_tracing=<optimized out>, kwdict=0x0, callargs=0x7f47a4fa5ea0, func=0x7f47a59dd760, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:7350
#78 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5377
#79 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7380, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#80 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#81 0x00007f47a6969bf0 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, args=args@entry=0x7fffc641bfa0, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0x7f47a4c489c0) at Objects/call.c:152
#82 0x00007f47a6969e84 in _PyObject_Call_Prepend (tstate=tstate@entry=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=callable@entry=0x7f47a6056020, obj=obj@entry=0x7f47a576ac50, args=args@entry=0x7f47a6d8bb18 <_PyRuntime+58904>, kwargs=kwargs@entry=0x7f47a4c489c0) at Objects/call.c:482
#83 0x00007f47a69e42c1 in slot_tp_call (self=0x7f47a576ac50, args=0x7f47a6d8bb18 <_PyRuntime+58904>, kwds=0x7f47a4c489c0) at Objects/typeobject.c:7630
#84 0x00007f47a69699f7 in _PyObject_MakeTpCall (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a576ac50, args=<optimized out>, nargs=0, keywords=0x7f47a5f4fd90) at Objects/call.c:214
#85 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#86 0x00007f47a6a75925 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df71b8, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#87 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7f47a643b880, func=0x7f47a62a2520, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Python/ceval.c:6428
#88 PyEval_EvalCode (co=co@entry=0x7f47a56eb440, globals=globals@entry=0x7f47a643b880, locals=locals@entry=0x7f47a643b880) at Python/ceval.c:1154
#89 0x00007f47a6a71740 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7f47a643b880, globals=0x7f47a643b880, source=0x7f47a56eb440) at Python/bltinmodule.c:1075
#90 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#91 0x00007f47a69bea4e in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f47a63d0fe0, args=0x7f47a6df7180, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:443
#92 0x00007f47a696a088 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7f47a63d0fe0, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#93 PyObject_Vectorcall (callable=0x7f47a63d0fe0, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:299
#94 0x00007f47a69088f3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4772
#95 0x00007f47a6a75a99 in _PyEval_EvalFrame (throwflag=0, frame=0x7f47a6df7020, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#96 _PyEval_Vector (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428
#97 0x00007f47a6969789 in _PyVectorcall_Call (kwargs=0x0, tuple=0x7f47a6241a00, callable=0x7f47a62a2200, func=0x7f47a6969900 <_PyFunction_Vectorcall>, tstate=0x7f47a6da5eb8 <_PyRuntime+166328>) at Objects/call.c:245
#98 _PyObject_Call (tstate=0x7f47a6da5eb8 <_PyRuntime+166328>, callable=0x7f47a62a2200, args=0x7f47a6241a00, kwargs=0x0) at Objects/call.c:328
#99 0x00007f47a6ae3239 in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at Modules/main.c:300
#100 0x00007f47a6ae3afa in pymain_run_python (exitcode=0x7fffc641c5b0) at Modules/main.c:595
#101 Py_RunMain () at Modules/main.c:680
#102 0x00007f47a6ae41ea in pymain_main (args=0x7fffc641c6d0) at Modules/main.c:710
#103 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734
#104 0x00007f47a6639083 in __libc_start_main (main=0x55d9d8329060 <main>, argc=5, argv=0x7fffc641c838, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffc641c828) at ../csu/libc-start.c:308
#105 0x000055d9d832909e in _start ()
mgeplf commented 2 months ago

My current theory is that: 1) Since there is a single invocation of the python interpreter, there are interactions between the tests being run 2) There is global state being stored, specifically sgid2srcindex_ and visources_ in src/nrniv/partrans.cpp 3) test/pytest_coreneuron/test_partrans.py cause the above globals to be modified - the Node's allocated and stored get free'd, but the references in the globals aren't cleared. 4) Non-deterministically, a later test overwrites some heap memory 5) Later, in mk_ttd, the dangling Node ref is used: https://github.com/neuronsimulator/nrn/blob/master/src/nrniv/partrans.cpp#L499) 6) Depending on the overwrite in 4), there is a segfault

When I have some more time, I'll see if I can confirm the theory

mgeplf commented 1 month ago

Got a valgrind trace that summarizes the above:

test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_swc.py ==21626== Invalid read of size 8
==21626==    at 0x8EDD150: mk_ttd() (partrans.cpp:503)
==21626==    by 0x8F49D26: nrn_thread_memblist_setup() (multicore.cpp:647)
==21626==    by 0x8F406DC: v_setup_vectors() (treeset.cpp:1697)
==21626==    by 0x8F37912: nrnhoc_topology() (solve.cpp:296)
==21626==    by 0x8F530E6: hoc_call() (code.cpp:1418)
==21626==    by 0x8FDFF19: fcall(void*, void*) (nrnpy_hoc.cpp:728)
==21626==    by 0x8ED9ACD: OcJump::fpycall(void* (*)(void*, void*), void*, void*) (ocjump.cpp:138)
==21626==    by 0x8FDCF09: hocobj_call(PyHocObject*, _object*, _object*) (nrnpy_hoc.cpp:796)
==21626==    by 0x49AC9F6: _PyObject_MakeTpCall (call.c:214)
==21626==    by 0x494B8F2: _PyEval_EvalFrameDefault (ceval.c:4772)
==21626==    by 0x4AB8A98: _PyEval_EvalFrame (pycore_ceval.h:73)
==21626==    by 0x4AB8A98: _PyEval_Vector (ceval.c:6428)
==21626==    by 0x49AC788: _PyVectorcall_Call (call.c:245)
==21626==    by 0x49AC788: _PyObject_Call (call.c:328)
==21626==  Address 0x8676a98 is 104 bytes inside a block of size 120 free'd
==21626==    at 0x483D1CF: operator delete(void*, unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==21626==    by 0x8F37B85: node_destruct(Node**, int) (solve.cpp:585)
==21626==    by 0x8F39597: node_free (solve.cpp:474)
==21626==    by 0x8F39597: sec_free(hoc_Item*) (solve.cpp:502)
==21626==    by 0x8F05D47: delete_section() (cabcode.cpp:358)
==21626==    by 0x8F530E6: hoc_call() (code.cpp:1418)
==21626==    by 0x8FDFF19: fcall(void*, void*) (nrnpy_hoc.cpp:728)
==21626==    by 0x8ED9ACD: OcJump::fpycall(void* (*)(void*, void*), void*, void*) (ocjump.cpp:138)
==21626==    by 0x8FDCF09: hocobj_call(PyHocObject*, _object*, _object*) (nrnpy_hoc.cpp:796)
==21626==    by 0x49AC9F6: _PyObject_MakeTpCall (call.c:214)
==21626==    by 0x494B8F2: _PyEval_EvalFrameDefault (ceval.c:4772)
==21626==    by 0x4AB8A98: _PyEval_EvalFrame (pycore_ceval.h:73)
==21626==    by 0x4AB8A98: _PyEval_Vector (ceval.c:6428)
==21626==    by 0x49AC788: _PyVectorcall_Call (call.c:245)
==21626==    by 0x49AC788: _PyObject_Call (call.c:328)
==21626==  Block was alloc'd at
==21626==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==21626==    by 0x8F37D48: node_alloc(Section*, short) (solve.cpp:744)
==21626==    by 0x8F02E30: nrn_change_nseg(Section*, int) (cabcode.cpp:1508)
==21626==    by 0x8F03977: new_section(Object*, Symbol*, int) (cabcode.cpp:298)
==21626==    by 0x8F0467F: nrnpy_newsection(NPySecObj*) (cabcode.cpp:327)
==21626==    by 0x8FE7A99: NPySecObj_init(NPySecObj*, _object*, _object*) (nrnpy_nrn.cpp:376)
==21626==    by 0x8FE7B8C: NPySecObj_new(_typeobject*, _object*, _object*) (nrnpy_nrn.cpp:401)
==21626==    by 0x4A013E2: cfunction_call (methodobject.c:542)
==21626==    by 0x49AC9F6: _PyObject_MakeTpCall (call.c:214)
==21626==    by 0x494B8F2: _PyEval_EvalFrameDefault (ceval.c:4772)
==21626==    by 0x4AB8A98: _PyEval_EvalFrame (pycore_ceval.h:73)
==21626==    by 0x4AB8A98: _PyEval_Vector (ceval.c:6428)
==21626==    by 0x49ACB7A: _PyObject_FastCallDictTstate (call.c:141)

The reduced set of tests required to trigger it somewhat often (1 in 10 times?) is:

python3 -m pytest \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_basic.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_bbss.py \
    \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_coreneuron_configuration.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_hoc_po.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_nrntest_fast.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_partrans.py \
    test/pytest_coreneuron/basic_tests_py3.11/test/pytest_coreneuron/test_swc.py \
pramodk commented 1 month ago

@mgeplf : I haven't looked at the code details but not surprised by the global state-related issue. With the code that you have skimmed through, do you know already a potential fix?

mgeplf commented 1 month ago

My guess is that one of node_destruct/node_free/sec_free/delete_section has to be made aware of partrans.cpp::visources_.