search

AllenInstitute / bmtk

Brain Modeling Toolkit
https://alleninstitute.github.io/bmtk/
BSD 3-Clause "New" or "Revised" License
272 stars 88 forks source link

pointnet: test_glif_utils.py Fatal Python error: Segmentation fault #274

Closed mitometa closed 1 year ago

mitometa commented 1 year ago

pytest error messages:

collected 358 items                                                                                                               

builder/test_connection_map.py .......                                                                                      [  1%]
builder/test_connector.py .....                                                                                             [  3%]
builder/test_densenetwork.py ......                                                                                         [  5%]
builder/test_edge_iterator.py ...                                                                                           [  5%]
builder/test_edges_sorter.py ...                                                                                            [  6%]
builder/test_id_generator.py ......                                                                                         [  8%]
builder/test_index_builders.py ..                                                                                           [  8%]
builder/test_iterator.py ........                                                                                           [ 11%]
builder/test_network.py ............                                                                                        [ 14%]
builder/test_network_mpi.py ssssssss                                                                                        [ 16%]
builder/test_node_pool.py .....                                                                                             [ 18%]
builder/test_node_set.py ...                                                                                                [ 18%]
builder/test_save_network.py .....                                                                                          [ 20%]
builder/auxi/test_edge_connectors.py ..                                                                                     [ 20%]
builder/auxi/test_node_params.py ......F                                                                                    [ 22%]
builder/bionet/test_swc_reader.py s                                                                                         [ 23%]
simulator/bionet/test_biograph.py ss                                                                                        [ 23%]
simulator/bionet/test_gids.py .                                                                                             [ 24%]
simulator/bionet/test_morphology.py .....s....                                                                              [ 26%]
simulator/bionet/test_nrn.py ....sss                                                                                        [ 28%]
simulator/bionet/test_virtualcell.py FFFFFFF..                                                                              [ 31%]
simulator/core/test_io_tools.py ..                                                                                          [ 31%]
simulator/core/test_pyfunction_cache.py ......                                                                              [ 33%]
simulator/core/test_simulation_config.py ..                                                                                 [ 34%]
simulator/core/test_simulation_config_validator.py ...                                                                      [ 34%]
simulator/filternet/test_default_setters.py ..................                                                              [ 39%]
simulator/filternet/lgnmodel/test_kernels.py ...                                                                            [ 40%]
simulator/filternet/lgnmodel/test_movie.py ...                                                                              [ 41%]
simulator/filternet/lgnmodel/test_subclass_metrics.py ......                                                                [ 43%]
simulator/filternet/lgnmodel/test_transferfunction.py .                                                                     [ 43%]
simulator/filternet/lgnmodel/test_units.py ....                                                                             [ 44%]
simulator/filternet/lgnmodel/test_utils.py .                                                                                [ 44%]
simulator/pointnet/test_glif_utils.py Fatal Python error: Segmentation fault

Current thread 0x00007fdc2ee61740 (most recent call first):
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/nest/ll_api.py", line 124 in catching_sli_run
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/nest/lib/hl_api_simulation.py", line 65 in Simulate
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/nest/ll_api.py", line 246 in stack_checker_func
  File "/home/jskang/softwares/bmtk/bmtk/tests/simulator/pointnet/test_glif_utils.py", line 38 in test_converter_psc
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/python.py", line 1789 in runtest
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 339 in from_call
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 324 in _main
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/usr/local/miniconda3/envs/brain/bin/pytest", line 10 in <module>
Segmentation fault (core dumped

platform:

linux -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
plugins: anyio-3.6.2, cov-4.0.0
nest-simulator            2.20.0 
neuron                    8.0.2
bmtk.__version__         '1.0.7'
mitometa commented 1 year ago

The bug is weird since I could run the examples (point_450cells and point_450glifs) smoothly.

In addition, the GLIF Network version of "Models of the Mouse Primary Visual Cortex V1" (http://portal.brain-map.org/explore/models/mv1-all-layers) also failed with this Segmentation fault after replacing the invalid inputs files with the input files of the Biophysical Network version.

xpliu16 commented 1 year ago

Can you get more info about which test fails e.g., by running pytest simulator/pointnet/test_glif_utils.py?

My intuition is that you have a fairly old version of NEST and a fairly new version of Python. I would suggest trying an environment with a newer Nest version. The 2.20.0 documentation (see link below) mentions at least one known bug (probably unrelated to this particular issue) and under "PyNEST" I see "Allow Installation with Python 3.8" but I don't see anything about 3.9.

https://github.com/nest/nest-simulator/releases/tag/v2.20.0

mitometa commented 1 year ago 
pytest simulator/pointnet/test_glif_utils.py
=================== test session starts ========================
platform linux -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
rootdir: /softwares/bmtk
plugins: anyio-3.6.2, cov-4.0.0
collected 5 items                                                                                                                     

simulator/pointnet/test_glif_utils.py .....                                                                                     [100%]

========================= 5 passed in 0.06s ================================

?

I will try a new one, thanks for the suggestion.

mitometa commented 1 year ago

The error was evoked by test_pointnetwork.py

pytest test_pointnetwork.py 
================= test session starts ====================
platform linux -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
rootdir: /softwares/bmtk
plugins: anyio-3.6.2, cov-4.0.0
collected 17 items                                                                                                                    

test_pointnetwork.py ....Fatal Python error: Aborted

Current thread 0x00007f7e73824740 (most recent call first):
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/nest/ll_api.py", line 124 in catching_sli_run
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/nest/lib/hl_api_connections.py", line 104 in GetConnections
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/nest/ll_api.py", line 246 in stack_checker_func
  File "/home/jskang/softwares/bmtk/bmtk/tests/simulator/pointnet/test_pointnetwork.py", line 43 in test_add_edges
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/python.py", line 1789 in runtest
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 339 in from_call
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 324 in _main
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/usr/local/miniconda3/envs/brain/lib/python3.9/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/usr/local/miniconda3/envs/brain/bin/pytest", line 10 in <module>
Aborted (core dumped)
mitometa commented 1 year ago

After the nest update to 3.3, pytest finished 100% with 8 failures:

==================== short test summary info ======================
FAILED tests/builder/auxi/test_node_params.py::test_positions_nrrd - FileNotFoundError: [Errno 2] No such file or directory: 'docs/tutorial/sources/nrrd/structure_721.nrrd'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[spike_times0] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[spike_times1] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[None] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[spike_times3] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[spike_times4] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[spike_times5] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
FAILED tests/simulator/bionet/test_virtualcell.py::test_spiketrain[spike_times6] - AttributeError: 'hoc.HocObject' object has no attribute 'VecStim'
=========== 8 failed, 332 passed, 18 skipped, 23 warnings in 18.92s =====================
xpliu16 commented 1 year ago

It seems you're running the tests from bmtk/bmtk - whereas I think the CI runs it from bmtk which contains also the docs. That way the relative path should makes sense for the top failed test. Could you try running from the "outer" bmtk directory? Please let me know if that helps and what tests still fail after that.

Thanks!

mitometa commented 1 year ago 
pytest
================================================== test session starts ==================================================
platform linux -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0
rootdir: /softwares/bmtk
plugins: anyio-3.6.2, cov-4.0.0
collecting ... Warning: no DISPLAY environment variable.
--No graphics will be displayed.
NEURON mechanisms not found in /softwares/bmtk/bmtk/tests/simulator/bionet/components/mechanisms.

              -- N E S T --
  Copyright (C) 2004 The NEST Initiative

 Version: 3.3
 Built: Nov  6 2022 20:14:32

 This program is provided AS IS and comes with
 NO WARRANTY. See the file LICENSE for details.

 Problems or suggestions?
   Visit https://www.nest-simulator.org

 Type 'nest.help()' to find out more about NEST.

collected 359 items / 30 errors                                                                                         

===================== ERRORS ===================
...

=================== short test summary info ===================
ERROR V1_Network_Models_from_the_Allen_Institute/Extracellular Field Potential/icsd_scripts/test_icsd.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_cellmodel.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_kernel.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_linearfilter.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_movie.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_multi_lnunit_multi_movie.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_poissongeneration.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/lgnmodel/lgnmodel/test/test_speedups.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/column_CC_matrix/test_CC_connections.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_basic.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_connect.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_io.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_nxnetwork.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_position.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_synnetwork.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/network2/network2/test/test_yazan.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_basic.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_cellmodel.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_kernel.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_linearfilter.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_movie.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_multi_lnunit_multi_movie.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_poissongeneration.py
ERROR V1_Network_Models_from_the_Allen_Institute/LGN Network/scripts/orig/lgnmodel/lgnmodel/test/test_speedups.py
ERROR V1_Network_Models_from_the_Allen_Institute/legacy_files/Biophysical_network/simulation_software/isee_engine/test/test_basic.py
ERROR V1_Network_Models_from_the_Allen_Institute/legacy_files/Biophysical_network/simulation_software/isee_engine/test/test_bionet.py
ERROR V1_Network_Models_from_the_Allen_Institute/legacy_files/Biophysical_network/simulation_software/isee_engine/test/test_nwb.py
ERROR V1_Network_Models_from_the_Allen_Institute/legacy_files/Biophysical_network/simulation_software/isee_engine/test/test_pointnet.py - FileNotFoundError: [Errno 2] No such file or directory: '/home/jungl/isee_engine/examples/run_pointnet'
ERROR V1_Network_Models_from_the_Allen_Institute/legacy_files/Biophysical_network/simulation_software/isee_engine/test/test_popnet.py
!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 30 errors during collection !!!!!!!!!!!!!!!!!!!!!!!
=============== 30 errors in 4.22s =======================

These errors are more related to the old files of V1 model, and the rest source codes of bmtk work well.

mitometa commented 1 year ago

Ping, all set, thanks!

mitometa commented 1 year ago

These errors of the V1 model could be confirmed after test one by one:

=========== short test summary info =============
ERROR lgnmodel/test/test_cellmodel.py
ERROR lgnmodel/test/test_kernel.py
ERROR lgnmodel/test/test_linearfilter.py
ERROR lgnmodel/test/test_movie.py
ERROR lgnmodel/test/test_multi_lnunit_multi_movie.py
ERROR lgnmodel/test/test_poissongeneration.py
ERROR lgnmodel/test/test_speedups.py
!!!!!!!!!!!!!!!!!!!!!! Interrupted: 7 errors during collection !!!!!!!!!!!!!!!!!!
xpliu16 commented 1 year ago

Unfortunately, this is from legacy code (from even before BMTK days) that is no longer supported. So you shouldn't be impacted by these test results. Let us know if there are any further issues!