Error: undefined symbol: _Z9nrnpy_hocv

jhielson commented 2 years ago

Hi,

I have successfully compiled and built Coreneuron on my laptop. I have also managed to compile my mod files (nrnivmodl -coreneuron .).

However, I get the following error (undefined symbol: _Z9nrnpy_hocv) every time I try to run my network. Can you help me?

Command

mpirun -n 6 ./x86_64/special -mpi -python completeModel-myOwnSTDP.py

Output:

--------------------------------------------------------------------------
[[2982,1],3]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: jhielson-Alienware-15-R4

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.1.0 HEAD (047dd824) 2022-03-25
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits

Additional mechanisms from files
 "./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"
Traceback (most recent call last):
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 135, in <module>
    from . import hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "completeModel-myOwnSTDP.py", line 2, in <module>
    from netpyne import specs, sim
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/__init__.py", line 21, in <module>
    from netpyne import analysis
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/__init__.py", line 17, in <module>
    from .spikes import prepareSpikeData, prepareRaster, prepareSpikeHist, popAvgRates
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/spikes.py", line 34, in <module>
    from ..specs import Dict
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/__init__.py", line 14, in <module>
    from .netParams import NetParams, CellParams
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/netParams.py", line 30, in <module>
    from .. import conversion
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/__init__.py", line 13, in <module>
    from .neuronPyHoc import importCell, importCellsFromNet, mechVarList, getSecName
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/neuronPyHoc.py", line 19, in <module>
    from neuron import h
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 137, in <module>
    import neuron.hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv
Traceback (most recent call last):
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 135, in <module>
    from . import hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "completeModel-myOwnSTDP.py", line 2, in <module>
    from netpyne import specs, sim
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/__init__.py", line 21, in <module>
    from netpyne import analysis
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/__init__.py", line 17, in <module>
    from .spikes import prepareSpikeData, prepareRaster, prepareSpikeHist, popAvgRates
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/spikes.py", line 34, in <module>
    from ..specs import Dict
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/__init__.py", line 14, in <module>
    from .netParams import NetParams, CellParams
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/netParams.py", line 30, in <module>
    from .. import conversion
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/__init__.py", line 13, in <module>
    from .neuronPyHoc import importCell, importCellsFromNet, mechVarList, getSecName
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/neuronPyHoc.py", line 19, in <module>
    from neuron import h
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 137, in <module>
    import neuron.hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv
Traceback (most recent call last):
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 135, in <module>
    from . import hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "completeModel-myOwnSTDP.py", line 2, in <module>
    from netpyne import specs, sim
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/__init__.py", line 21, in <module>
    from netpyne import analysis
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/__init__.py", line 17, in <module>
    from .spikes import prepareSpikeData, prepareRaster, prepareSpikeHist, popAvgRates
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/spikes.py", line 34, in <module>
    from ..specs import Dict
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/__init__.py", line 14, in <module>
    from .netParams import NetParams, CellParams
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/netParams.py", line 30, in <module>
    from .. import conversion
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/__init__.py", line 13, in <module>
    from .neuronPyHoc import importCell, importCellsFromNet, mechVarList, getSecName
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/neuronPyHoc.py", line 19, in <module>
    from neuron import h
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 137, in <module>
    import neuron.hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv
Traceback (most recent call last):
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 135, in <module>
    from . import hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "completeModel-myOwnSTDP.py", line 2, in <module>
    from netpyne import specs, sim
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/__init__.py", line 21, in <module>
    from netpyne import analysis
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/__init__.py", line 17, in <module>
    from .spikes import prepareSpikeData, prepareRaster, prepareSpikeHist, popAvgRates
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/spikes.py", line 34, in <module>
    from ..specs import Dict
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/__init__.py", line 14, in <module>
    from .netParams import NetParams, CellParams
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/netParams.py", line 30, in <module>
    from .. import conversion
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/__init__.py", line 13, in <module>
    from .neuronPyHoc import importCell, importCellsFromNet, mechVarList, getSecName
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/neuronPyHoc.py", line 19, in <module>
    from neuron import h
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 137, in <module>
    import neuron.hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv
Traceback (most recent call last):
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 135, in <module>
    from . import hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "completeModel-myOwnSTDP.py", line 2, in <module>
    from netpyne import specs, sim
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/__init__.py", line 21, in <module>
    from netpyne import analysis
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/__init__.py", line 17, in <module>
    from .spikes import prepareSpikeData, prepareRaster, prepareSpikeHist, popAvgRates
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/spikes.py", line 34, in <module>
    from ..specs import Dict
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/__init__.py", line 14, in <module>
    from .netParams import NetParams, CellParams
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/netParams.py", line 30, in <module>
    from .. import conversion
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/__init__.py", line 13, in <module>
    from .neuronPyHoc import importCell, importCellsFromNet, mechVarList, getSecName
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/neuronPyHoc.py", line 19, in <module>
    from neuron import h
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 137, in <module>
    import neuron.hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv
Traceback (most recent call last):
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 135, in <module>
    from . import hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "completeModel-myOwnSTDP.py", line 2, in <module>
    from netpyne import specs, sim
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/__init__.py", line 21, in <module>
    from netpyne import analysis
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/__init__.py", line 17, in <module>
    from .spikes import prepareSpikeData, prepareRaster, prepareSpikeHist, popAvgRates
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/analysis/spikes.py", line 34, in <module>
    from ..specs import Dict
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/__init__.py", line 14, in <module>
    from .netParams import NetParams, CellParams
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/specs/netParams.py", line 30, in <module>
    from .. import conversion
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/__init__.py", line 13, in <module>
    from .neuronPyHoc import importCell, importCellsFromNet, mechVarList, getSecName
  File "/home/jhielson/.local/lib/python3.8/site-packages/netpyne/conversion/neuronPyHoc.py", line 19, in <module>
    from neuron import h
  File "/home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/__init__.py", line 137, in <module>
    import neuron.hoc
ImportError: /home/jhielson/coreneuron_repository/nrn/build/lib/python/neuron/hoc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z9nrnpy_hocv
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[2982,1],3]
  Exit code:    1
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:67411] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:67411] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

iomaganaris commented 2 years ago

Hello @jhielson ,

I looked into this but I'm not entirely sure what is the issue exactly yet. Could you check whether NRN_ENABLE_PYTHON_DYNAMIC is disabled in your NEURON cmake build? The only way I could reproduce it is by having it enabled while the default is disabled. I'll come back to you when I have more details about this issue

jhielson commented 2 years ago

Thanks!!! I believe that might be the problem. It was off when I built the package.

-- Python        | ON
--   EXE         | /usr/bin/python3
--   INC         | /usr/include/python3.8
--   LIB         | /usr/lib/x86_64-linux-gnu/libpython3.8.so
--   MODULE      | ON
--   DYNAMIC     | OFF

I don't have my laptop with me right now to test it but I'll try that as soon I return to the lab. And I will let you know here. Thank you again.

jhielson commented 2 years ago

Dear @iomaganaris,

Sorry, I believe I have misunderstood your message. It was disabled the whole time as you can see below:

Command:


cmake .. -DNRN_ENABLE_CORENEURON=ON -DCORENRN_ENABLE_GPU=ON -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_RX3D=OFF -DCMAKE_INSTALL_PREFIX=$HOME/install -DCMAKE_C_COMPILER=nvc -DCMAKE_CXX_COMPILER=nvc++ -DCMAKE_CUDA_COMPILER=nvcc

Output:

-- The C compiler identification is PGI 22.3.0
-- The CXX compiler identification is PGI 22.3.0
-- Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc
-- Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting build type to 'RelWithDebInfo' as none was specified.
-- 3rd party project: using Random123 from "external/Random123"
-- Found BISON: /usr/bin/bison (found version "3.5.1") 
-- Found FLEX: /usr/bin/flex (found version "2.6.4") 
-- Found Readline: /usr/include  
-- Found MPI_C: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Detected OpenMPI 3.1.5
-- -DPYTHON_EXECUTABLE not specified. Looking for `python3` in the PATH exclusively...
--  Setting PYTHON_EXECUTABLE=/usr/bin/python3
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.10", minimum required is "3.7") 
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.10", minimum required is "3") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Building CoreNEURON from submodule
-- Sub-project : using coreneuron from from /home/jhielson/coreneuron_repository/nrn/external/coreneuron
-- Found HpcCodingConv: /home/jhielson/coreneuron_repository/nrn/external/coreneuron/CMake/hpc-coding-conventions  
-- CORENRN_FORMATTING: OFF
-- CORENRN_TEST_FORMATTING: OFF
-- CORENRN_CLANG_FORMAT: OFF
-- CORENRN_CMAKE_FORMAT: OFF
-- CORENRN_GIT_HOOKS: OFF
-- CORENRN_GIT_COMMIT_HOOKS: 
-- CORENRN_GIT_PUSH_HOOKS: courtesy-msg
-- CORENRN_STATIC_ANALYSIS: OFF
-- CORENRN_TEST_STATIC_ANALYSIS: OFF
-- Found Random123: /home/jhielson/coreneuron_repository/nrn/external/coreneuron/external/Random123  
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- 3rd party project: using CLI11 from "external/CLI11"
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Doxygen not found, building docs has been disabled
-- Setting default CUDA architectures to 70;80
-- The CUDA compiler identification is NVIDIA 11.6.112
-- Check for working CUDA compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvcc
-- Check for working CUDA compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Found PythonInterp: /usr/bin/python3 (found version "3.8.10") 
-- Found Perl: /usr/bin/perl (found version "5.30.0") 
-- Found MOD2C: /home/jhielson/coreneuron_repository/nrn/external/coreneuron/external/mod2c  
-- Using mod2c submodule from /home/jhielson/coreneuron_repository/nrn/external/coreneuron/external/mod2c
-- mod2c is used as APPLICATION_NAME
-- Boost found, enabling use of memory pools for Random123...
-- 
-- Configured CoreNEURON 8.2.0
-- 
-- You can now build CoreNEURON using:
--   cmake --build . --parallel 8 [--target TARGET]
-- You might want to adjust the number of parallel build jobs for your system.
-- Some non-default targets you might want to build:
-- --------------------+--------------------------------------------------------
--  Target             |   Description
-- --------------------+--------------------------------------------------------
-- install             | Will install CoreNEURON to: /home/jhielson/install
-- docs                | Build full docs. Calls targets: doxygen, sphinx
-- --------------------+--------------------------------------------------------
--  Build option       | Status
-- --------------------+--------------------------------------------------------
-- CXX COMPILER        | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++
-- COMPILE FLAGS       |   -g  -O2  --c++14 -cuda -gpu=cuda11.6,lineinfo,cc70,cc80 -acc -Mautoinline -DEIGEN_DONT_VECTORIZE=1
-- Build Type          | STATIC
-- MPI                 | ON
--   DYNAMIC           | OFF
--   INC               | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/include
-- OpenMP              | ON
-- Use legacy units    | OFF
-- NMODL               | OFF
-- MOD2CPP PATH        | /home/jhielson/coreneuron_repository/nrn/build/bin/mod2c_core
-- GPU Support         | ON
--   CUDA              | 
--   Offload           | OpenACC
--   Unified Memory    | OFF
-- Auto Timeout        | ON
-- Wrap exp()          | OFF
-- SplayTree Queue     | ON
-- NetReceive Buffer   | ON
-- Caliper             | OFF
-- Likwid              | OFF
-- Unit Tests          | OFF
-- Reporting           | OFF
-- --------------+--------------------------------------------------------------
--  See documentation : https://github.com/BlueBrain/CoreNeuron/
-- --------------+--------------------------------------------------------------
-- 
Extracting link flags from target 'Threads::Threads', beware that this can be fragile. Got: 
Generating link flags from path /usr/lib/x86_64-linux-gnu/libreadline.so Got: /usr/lib/x86_64-linux-gnu/libreadline.so -Wl,-rpath,/usr/lib/x86_64-linux-gnu
Generating link flags from path /usr/lib/x86_64-linux-gnu/libpython3.8.so Got: /usr/lib/x86_64-linux-gnu/libpython3.8.so -Wl,-rpath,/usr/lib/x86_64-linux-gnu
Generating link flags from path /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so Got: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so -Wl,-rpath,/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib
-- 
-- Configured NEURON 8.2.0
-- 
-- You can now build NEURON using:
--   cmake --build . --parallel 8 [--target TARGET]
-- You might want to adjust the number of parallel build jobs for your system.
-- Some non-default targets you might want to build:
-- --------------+--------------------------------------------------------------
--  Target       |   Description
-- --------------+--------------------------------------------------------------
-- install       | Will install NEURON to: /home/jhielson/install
--               | Change the install location of NEURON using:
--               |   cmake <src_path> -DCMAKE_INSTALL_PREFIX=<install_path>
-- docs          | Build full docs. Calls targets: doxygen, notebooks, sphinx, notebooks-clean
-- uninstall     | Removes files installed by make install (todo)
-- --------------+--------------------------------------------------------------
--  Build option | Status
-- --------------+--------------------------------------------------------------
-- C COMPILER    | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc
-- CXX COMPILER  | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++
-- BUILD_TYPE    | RelWithDebInfo (allowed: Custom;Debug;Release;RelWithDebInfo;Fast)
-- COMPILE FLAGS | -g  -O2  --diag_suppress=1,47,111,128,170,174,177,180,186,301,541,550,816,941,2465
-- Shared        | ON
-- Default units | modern units (2019 nist constants)
-- MPI           | ON
--   DYNAMIC     | OFF
--   INC         | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/include
--   LIB         | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so
-- Python        | ON
--   EXE         | /usr/bin/python3
--   INC         | /usr/include/python3.8
--   LIB         | /usr/lib/x86_64-linux-gnu/libpython3.8.so
--   MODULE      | ON
--   DYNAMIC     | OFF
-- Readline      | /usr/lib/x86_64-linux-gnu/libreadline.so
-- RX3D          | OFF
-- Interviews    | OFF
-- CoreNEURON    | ON
--   PATH        | /home/jhielson/coreneuron_repository/nrn/external/coreneuron
--   LINK FLAGS  | -cuda -gpu=cuda11.6,lineinfo,cc70,cc80 -acc -rdynamic -lrt -Wl,--whole-archive -Lx86_64 -lcorenrnmech -L$(libdir) -lcoreneuron -Wl,--no-whole-archive /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so
--   Legacy Units| OFF
-- Tests         | OFF
-- --------------+--------------------------------------------------------------
--  See documentation : https://www.neuron.yale.edu/neuron/
-- --------------+--------------------------------------------------------------
-- 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jhielson/coreneuron_repository/nrn/build

iomaganaris commented 2 years ago

Hello @jhielson ,

It's disabled as it should be for now. Could you try prepending the NEURON python package installation directory to your PYTHONPATH and trying launching the special again? You can do this by running the following before your script:

export PYTHONPATH=$HOME/install/lib/python:$PYTHONPATH

Also it could be useful to share the output of the following commands:

echo $PYTHONPATH
ldd $HOME/install/lib/python/neuron/hoc.cpython-*.so
nm $HOME/install/lib/libnrniv.so

Thank you very much

jhielson commented 2 years ago

Ok, I am gonna do that and let you know here. I have been testing some previous versions of NVHPC but got the same issue.

jhielson commented 2 years ago

Hi,

I exported the python path as suggested and removed the "x86_64" folder. Then, I recompiled my mods file (nrnivmodl -coreneuron .). Now, I get the following error:

No such file or directory: '/home/jhielson/install/lib/python/neuron/.data/bin/nrnivmodl'

Command:

echo $PYTHONPATH

Output:

/home/jhielson/install/lib/python:/home/jhielson/coreneuron_repository/nrn/build/lib/python:/opt/ros/noetic/lib/python3/dist-packages

Command:

ldd $HOME/install/lib/python/neuron/hoc.cpython-*.so

Output:

linux-vdso.so.1 (0x00007ffd707d0000)
    libnrniv.so => /home/jhielson/install/lib/libnrniv.so (0x00007f5cc5294000)
    libatomic.so.1 => /usr/lib/x86_64-linux-gnu/libatomic.so.1 (0x00007f5cc5240000)
    libnvhpcatm.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvhpcatm.so (0x00007f5cc5035000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5cc4e53000)
    libnvomp.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvomp.so (0x00007f5cc4151000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5cc4149000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5cc4126000)
    libnvcpumath.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvcpumath.so (0x00007f5cc3ce5000)
    libnvc.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvc.so (0x00007f5cc3a84000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5cc3892000)
    libgcc_s.so.1 => /../lib/libgcc_s.so.1 (0x00007f5cc3877000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5cc3726000)
    libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007f5cc36d6000)
    libpython3.8.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f5cc3180000)
    libmpi.so.40 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so.40 (0x00007f5cc2cf8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5cc57b6000)
    libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f5cc2cc8000)
    libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f5cc2c98000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5cc2c7c000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5cc2c77000)
    libopen-rte.so.40 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libopen-rte.so.40 (0x00007f5cc2912000)
    libopen-pal.so.40 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libopen-pal.so.40 (0x00007f5cc23f0000)
    librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f5cc23d1000)
    libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f5cc23b2000)
    libnuma.so.1 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libnuma.so.1 (0x00007f5cc21a7000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5cc219d000)
    libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f5cc2178000)
    libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f5cc2100000)

Command:

nm $HOME/install/lib/libnrniv.so

The output of this one was too long. So I changed the command to:

Command:

nm $HOME/install/lib/libnrniv.so | grep _Z9nrnpy_hocv

Output:

00000000003f7840 T _Z9nrnpy_hocv

jhielson commented 2 years ago

Dear @iomaganaris,

I have reinstalled coreneuron and now I got a different error:

mpirun -n 6 ./x86_64/special -mpi -python completeModel-myOwnSTDP.py 
--------------------------------------------------------------------------
[[40883,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: jhielson-Alienware-15-R4

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.2a-5-g6e4f8cb0+ master (6e4f8cb0+) 2022-06-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits

Additional mechanisms from files
 "./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"

Start time:  2022-06-20 12:53:29.684249

Creating network of 674 cell populations on 6 hosts...
  Number of cells on node 1: 138 
  Number of cells on node 2: 138 
  Number of cells on node 3: 138 
  Number of cells on node 4: 138 
  Number of cells on node 5: 138 
  Number of cells on node 0: 138 
  Done; cell creation time = 0.24 s.
Making connections...
  Number of connections on node 5: 1040 
  Number of synaptic contacts on node 5: 1046 
  Number of connections on node 0: 1061 
  Number of synaptic contacts on node 0: 1069 
[jhielson-Alienware-15-R4:37703] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:37703] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
  Number of connections on node 4: 1647 
  Number of synaptic contacts on node 4: 1653 
  Number of connections on node 1: 1682 
  Number of synaptic contacts on node 1: 1688 
  Number of connections on node 2: 1683 
  Number of synaptic contacts on node 2: 1689 
  Number of connections on node 3: 1658 
  Number of synaptic contacts on node 3: 1664 
  Done; cell connection time = 3.27 s.
Adding stims...
  Number of stims on node 4: 30 
  Number of stims on node 1: 31 
  Number of stims on node 0: 30 
  Number of stims on node 2: 31 
  Number of stims on node 3: 30 
  Number of stims on node 5: 30 
  Done; cell stims creation time = 0.04 s.
exp(764.628) out of range, returning exp(700)
exp(764.628) out of range, returning exp(700)
exp(834.429) out of range, returning exp(700)
exp(808.336) out of range, returning exp(700)
exp(808.336) out of range, returning exp(700)
exp(727.905) out of range, returning exp(700)
exp(727.905) out of range, returning exp(700)
exp(796.118) out of range, returning exp(700)
exp(796.118) out of range, returning exp(700)
exp(814.896) out of range, returning exp(700)
exp(814.896) out of range, returning exp(700)
exp(834.429) out of range, returning exp(700)

Running with interval func using CoreNEURON for 300000.0 ms...
5 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
5  near line 0
5  tstop=300000.0
   3 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
3  near line 0
3  tstop=300000.0
      2 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
1 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
1  near line 0
1  tstop=300000.0
           4 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
4  near line 0
4  tstop=300000.0
             0 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
0  near line 0
0  tstop=300000.0
                  ^
               ^
        3 ParallelContext[5].psolve(    ^
        1 ParallelContext[5].psolve(1)
  ^
        4 ParallelContext[5].psolve(1)
         ^
        0 ParallelContext[5].psolve(1)
  5 ParallelContext[5].psolve(1)
1)
2  near line 0
2  tstop=300000.0
               ^
        2 ParallelContext[5].psolve(1)
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:37703] 5 more processes have sent help message help-mpi-api.txt / mpi-abort

Basically, it says:

NEURON model for CoreNEURON requires cvode.cache_efficient(1)

Can you help me with this new issue?

iomaganaris commented 2 years ago

Hello @jhielson,

Nice. It looks like we're making some progress. Regarding your new issue and assuming that you're using NetPyNE, can you make sure that in your NetPyNE SimConfig the variable cache_efficient is True? You can enable it by setting the following in your cfg.py:

cfg.cache_efficient = True

jhielson commented 2 years ago

Thanks. Now, it says it can not find GPU.

--------------------------------------------------------------------------
[[38348,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: jhielson-Alienware-15-R4

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.2a-5-g6e4f8cb0+ master (6e4f8cb0+) 2022-06-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits

Additional mechanisms from files
 "./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"

Start time:  2022-06-20 13:01:36.929230

Creating network of 149 cell populations on 6 hosts...
  Number of cells on node 1: 51 
  Number of cells on node 2: 51 
  Number of cells on node 3: 50 
  Number of cells on node 4: 50 
  Number of cells on node 5: 50 
  Number of cells on node 0: 51 
  Done; cell creation time = 0.05 s.
Making connections...
  Number of connections on node 3: 452 
  Number of synaptic contacts on node 3: 460 
  Number of connections on node 1: 510 
  Number of synaptic contacts on node 1: 516 
  Number of connections on node 4: 542 
  Number of synaptic contacts on node 4: 548 
  Number of connections on node 0: 523 
  Number of synaptic contacts on node 0: 529 
  Number of connections on node 2: 430 
  Number of synaptic contacts on node 2: 436 
  Number of connections on node 5: 541 
  Number of synaptic contacts on node 5: 547 
  Done; cell connection time = 0.59 s.
Adding stims...
  Number of stims on node 0: 30 
  Number of stims on node 4: 31 
  Number of stims on node 5: 31 
  Number of stims on node 1: 30 
  Number of stims on node 2: 30 
  Number of stims on node 3: 30 
  Done; cell stims creation time = 0.02 s.
exp(751.718) out of range, returning exp(700)
exp(751.718) out of range, returning exp(700)
exp(823.67) out of range, returning exp(700)
exp(823.67) out of range, returning exp(700)
exp(874.045) out of range, returning exp(700)
exp(874.045) out of range, returning exp(700)
exp(720.81) out of range, returning exp(700)
exp(720.81) out of range, returning exp(700)
exp(732.031) out of range, returning exp(700)
exp(732.031) out of range, returning exp(700)
exp(823.791) out of range, returning exp(700)
exp(823.791) out of range, returning exp(700)

Running with interval func using CoreNEURON for 20000.0 ms...
 num_mpi=6

 ERROR : Enabled GPU execution but couldn't find NVIDIA GPU!
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:39224] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:39224] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[jhielson-Alienware-15-R4:39224] 5 more processes have sent help message help-mpi-api.txt / mpi-abort

But I have one working fine.

Command:

nvidia-smi

Output:

Mon Jun 20 13:03:27 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| N/A   71C    P0    47W /  N/A |   1418MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1136      G   /usr/lib/xorg/Xorg                150MiB |
|    0   N/A  N/A      1717      G   /usr/lib/xorg/Xorg                583MiB |
|    0   N/A  N/A      1847      G   /usr/bin/gnome-shell              162MiB |
|    0   N/A  N/A      2557      G   /usr/lib/firefox/firefox          284MiB |
|    0   N/A  N/A     14545      G   ...RendererForSitePerProcess      222MiB |
+-----------------------------------------------------------------------------+

iomaganaris commented 2 years ago

Hello @jhielson,

What is the GPU model you're trying to execute CoreNEURON on? It might be that CoreNEURON is not built for your GPU architecture and this is why CoreNEURON cannot find it. If you find the Compute Capability for your GPU you can build CoreNEURON for it using the following CMake variable:

-DCMAKE_CUDA_ARCHITECTURES=61

for Compute Architecture 6.1 for example.

jhielson commented 2 years ago

Thanks. I have the following device:

description: VGA compatible controller
       product: GP104BM [GeForce GTX 1080 Mobile]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
       configuration: driver=nvidia latency=0
       resources: irq:163 memory:9c000000-9cffffff memory:70000000-7fffffff memory:80000000-81ffffff ioport:5000(size=128) memory:c0000-dffff

The compute capability seems to be 6.1.

I am going to compile coreneuron again and I will keep you informed.

jhielson commented 2 years ago

Dear @iomaganaris,

Now, it is recognizing the GPU but there is a new error: Command:

mpirun -n 6 ./x86_64/special -mpi -python completeModel-100GC.py

Output:

--------------------------------------------------------------------------
[[49409,1],2]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: jhielson-Alienware-15-R4

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.2a-5-g6e4f8cb0+ master (6e4f8cb0+) 2022-06-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits

Additional mechanisms from files
 "./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"

Start time:  2022-06-20 14:42:52.738020

Creating network of 149 cell populations on 6 hosts...
  Number of cells on node 1: 51 
  Number of cells on node 2: 51 
  Number of cells on node 3: 50 
  Number of cells on node 4: 50 
  Number of cells on node 5: 50 
  Number of cells on node 0: 51 
  Done; cell creation time = 0.08 s.
Making connections...
  Number of connections on node 2: 428 
  Number of synaptic contacts on node 2: 434 
  Number of connections on node 5: 545 
  Number of synaptic contacts on node 5: 551 
  Number of connections on node 3: 454 
  Number of synaptic contacts on node 3: 462 
  Number of connections on node 0: 525 
  Number of synaptic contacts on node 0: 531 
  Number of connections on node 4: 542 
  Number of synaptic contacts on node 4: 548 
  Number of connections on node 1: 506 
  Number of synaptic contacts on node 1: 512 
  Done; cell connection time = 0.54 s.
Adding stims...
  Number of stims on node 4: 31 
  Number of stims on node 2: 30 
  Number of stims on node 5: 31 
  Number of stims on node 3: 30 
  Number of stims on node 0: 30 
  Number of stims on node 1: 30 
  Done; cell stims creation time = 0.02 s.
exp(760.937) out of range, returning exp(700)
exp(760.937) out of range, returning exp(700)
exp(930.274) out of range, returning exp(700)
exp(930.274) out of range, returning exp(700)
exp(762.501) out of range, returning exp(700)
exp(762.501) out of range, returning exp(700)
exp(810.312) out of range, returning exp(700)
exp(810.312) out of range, returning exp(700)
exp(871.081) out of range, returning exp(700)
exp(871.081) out of range, returning exp(700)
exp(829.235) out of range, returning exp(700)
exp(829.235) out of range, returning exp(700)

Running with interval func using CoreNEURON for 20000.0 ms...
 num_mpi=6

 Info : 1 GPUs shared by 6 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 8.2.0 6215225f (2022-06-08 10:09:55 +0200)

 Additional mechanisms from files
 GP.mod Izhi2003b.mod STN2.mod Str.mod SynExp2NMDA.mod exp2syn.mod expsyn.mod fi_stdp.mod hh.mod izhi2007a.mod izhi2007bS.mod mySTDP.mod netstim.mod passive.mod pattern.mod spikeout.mod stdwa_songabbott.mod stim.mod svclmp.mod thalamus.mod

 Memory (MBs) :             After mk_mech : Max 398.7812, Min 398.0742, Avg 398.4473 
 GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
 Memory (MBs) :            After MPI_Init : Max 398.7812, Min 398.0742, Avg 398.4473 
 GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
 Memory (MBs) :          Before nrn_setup : Max 399.2930, Min 398.5078, Avg 398.8932 
 GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
 Setup Done   : 0.00 seconds 
 Model size   : 766.18 kB
 Memory (MBs) :          After nrn_setup  : Max 399.2930, Min 398.5078, Avg 398.8932 
 GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
GENERAL PARAMETERS
--mpi=true
--mpi-lib=
--gpu=true
--dt=0.1
--tstop=1

GPU
--nwarp=65536
--cell-permute=2
--cuda-interface=false

INPUT PARAMETERS
--voltage=1000
--seed=-1
--datpath=.
--filesdat=files.dat
--pattern=
--report-conf=
--restore=     

PARALLEL COMPUTATION PARAMETERS
--threading=false
--skip_mpi_finalize=true

SPIKE EXCHANGE
--ms_phases=2
--ms_subintervals=2
--multisend=false
--spk_compress=0
--binqueue=false

CONFIGURATION
--spikebuf=100000
--prcellgid=-1
--forwardskip=0
--celsius=6.3
--mindelay=1
--report-buffer-size=4

OUTPUT PARAMETERS
--dt_io=0.1
--outpath=.
--checkpoint=

 Start time (t) = 0

 Memory (MBs) :  After mk_spikevec_buffer : Max 399.2930, Min 398.5078, Avg 398.8932 
 GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
 Memory (MBs) :     After nrn_finitialize : Max 399.9531, Min 399.1055, Avg 399.5358 
 GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500

psolve |=========================================================| t: 1.00   ETA: 0h00m01s

Solver Time : 0.100294

 Simulation Statistics
 Number of cells: 282
 Number of compartments: 846
 Number of presyns: 481
 Number of input presyns: 1137
 Number of synapses: 3058
 Number of point processes: 3419
 Number of transfer sources: 0
 Number of transfer targets: 0
 Number of spikes: 12
 Number of spikes with non negative gid-s: 12
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
[jhielson-Alienware-15-R4:52731] *** Process received signal ***
[jhielson-Alienware-15-R4:52731] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52731] Signal code:  (-6)
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
[jhielson-Alienware-15-R4:52733] *** Process received signal ***
[jhielson-Alienware-15-R4:52733] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52733] Signal code:  (-6)
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
[jhielson-Alienware-15-R4:52730] *** Process received signal ***
[jhielson-Alienware-15-R4:52730] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52730] Signal code:  (-6)
[jhielson-Alienware-15-R4:52734] *** Process received signal ***
[jhielson-Alienware-15-R4:52734] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52734] Signal code:  (-6)
[jhielson-Alienware-15-R4:52732] *** Process received signal ***
[jhielson-Alienware-15-R4:52732] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52732] Signal code:  (-6)
[jhielson-Alienware-15-R4:52729] *** Process received signal ***
[jhielson-Alienware-15-R4:52729] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52729] Signal code:  (-6)
[jhielson-Alienware-15-R4:52731] [ 0] [jhielson-Alienware-15-R4:52734] [ 0] [jhielson-Alienware-15-R4:52733] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fe493b6a420]
[jhielson-Alienware-15-R4:52731] [ 1] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f326cdec420]
[jhielson-Alienware-15-R4:52733] [ 1] [jhielson-Alienware-15-R4:52729] [ 0] [jhielson-Alienware-15-R4:52732] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f65d5fdf420]
[jhielson-Alienware-15-R4:52732] [ 1] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f2442676420]
[jhielson-Alienware-15-R4:52734] [ 1] [jhielson-Alienware-15-R4:52730] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f5fd03a0420]
[jhielson-Alienware-15-R4:52730] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fe49330500b]
[jhielson-Alienware-15-R4:52731] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fe4932e4859]
[jhielson-Alienware-15-R4:52731] [ 3] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f326c58700b]
[jhielson-Alienware-15-R4:52733] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f326c566859]
[jhielson-Alienware-15-R4:52733] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f095c9b7420]
[jhielson-Alienware-15-R4:52729] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f095c15200b]
[jhielson-Alienware-15-R4:52729] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f65d577a00b]
[jhielson-Alienware-15-R4:52732] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f2441e1100b]
[jhielson-Alienware-15-R4:52734] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f2441df0859]
[jhielson-Alienware-15-R4:52734] [ 3] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f5fcfb3b00b]
[jhielson-Alienware-15-R4:52730] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7fe4932e4729]
[jhielson-Alienware-15-R4:52731] [ 4] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f2441df0729]
[jhielson-Alienware-15-R4:52734] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f2441e01fd6]
[jhielson-Alienware-15-R4:52734] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f5fcfb1a859]
[jhielson-Alienware-15-R4:52730] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f5fcfb1a729]
[jhielson-Alienware-15-R4:52730] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f5fcfb2bfd6]
[jhielson-Alienware-15-R4:52730] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7fe4932f5fd6]
[jhielson-Alienware-15-R4:52731] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f095c131859]
[jhielson-Alienware-15-R4:52729] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f095c131729]
[jhielson-Alienware-15-R4:52729] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f095c142fd6]
[jhielson-Alienware-15-R4:52729] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f65d5759859]
[jhielson-Alienware-15-R4:52732] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f65d5759729]
[jhielson-Alienware-15-R4:52732] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f65d576afd6]
[jhielson-Alienware-15-R4:52732] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f326c566729]
[jhielson-Alienware-15-R4:52733] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f326c577fd6]
[jhielson-Alienware-15-R4:52733] [ 5] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7fe4969b2e9e]
[jhielson-Alienware-15-R4:52731] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f24454bee9e]
[jhielson-Alienware-15-R4:52734] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f326fc34e9e]
[jhielson-Alienware-15-R4:52733] [ 6] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52731] [ 7] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f095f7ffe9e]
[jhielson-Alienware-15-R4:52729] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f65d8e27e9e]
[jhielson-Alienware-15-R4:52732] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f5fd31e8e9e]
[jhielson-Alienware-15-R4:52730] [ 6] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52733] [ 7] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52734] [ 7] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52731] [ 8] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52729] [ 7] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52730] [ 7] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52732] [ 7] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52734] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52731] [ 9] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52733] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52734] [ 9] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52729] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52733] [ 9] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52730] [ 8] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52732] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52729] [ 9] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52730] [ 9] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52732] [ 9] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7fe4969ad929]
[jhielson-Alienware-15-R4:52731] [10] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f24454b9929]
[jhielson-Alienware-15-R4:52734] [10] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f326fc2f929]
[jhielson-Alienware-15-R4:52733] [10] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f095f7fa929]
[jhielson-Alienware-15-R4:52729] [10] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7fe4969d2c91]
[jhielson-Alienware-15-R4:52731] [11] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f65d8e22929]
[jhielson-Alienware-15-R4:52732] [10] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f24454dec91]
[jhielson-Alienware-15-R4:52734] [11] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f326fc54c91]
[jhielson-Alienware-15-R4:52733] [11] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f095f81fc91]
[jhielson-Alienware-15-R4:52729] [11] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f5fd31e3929]
[jhielson-Alienware-15-R4:52730] [10] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f65d8e47c91]
[jhielson-Alienware-15-R4:52732] [11] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7fe496a6738e]
[jhielson-Alienware-15-R4:52731] [12] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f326fce938e]
[jhielson-Alienware-15-R4:52733] [12] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f244557338e]
[jhielson-Alienware-15-R4:52734] [12] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f095f8b438e]
[jhielson-Alienware-15-R4:52729] [12] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f5fd3208c91]
[jhielson-Alienware-15-R4:52730] [11] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7fe496bbc0bb]
[jhielson-Alienware-15-R4:52731] [13] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f65d8edc38e]
[jhielson-Alienware-15-R4:52732] [12] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f095fa090bb]
[jhielson-Alienware-15-R4:52729] [13] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f326fe3e0bb]
[jhielson-Alienware-15-R4:52733] [13] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f24456c80bb]
[jhielson-Alienware-15-R4:52734] [13] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7fe4969d66af]
[jhielson-Alienware-15-R4:52731] [14] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f5fd329d38e]
[jhielson-Alienware-15-R4:52730] [12] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f65d90310bb]
[jhielson-Alienware-15-R4:52732] [13] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f095f8236af]
[jhielson-Alienware-15-R4:52729] [14] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f326fc586af]
[jhielson-Alienware-15-R4:52733] [14] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f24454e26af]
[jhielson-Alienware-15-R4:52734] [14] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7fe496bbc772]
[jhielson-Alienware-15-R4:52731] [15] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f5fd33f20bb]
[jhielson-Alienware-15-R4:52730] [13] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7fe496485b1b]
[jhielson-Alienware-15-R4:52731] [16] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f095fa09772]
[jhielson-Alienware-15-R4:52729] [15] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f65d8e4b6af]
[jhielson-Alienware-15-R4:52732] [14] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7fe496251df3]
[jhielson-Alienware-15-R4:52731] [17] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f326fe3e772]
[jhielson-Alienware-15-R4:52733] [15] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f24456c8772]
[jhielson-Alienware-15-R4:52734] [15] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7fe496259ef6]
[jhielson-Alienware-15-R4:52731] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f095f2d2b1b]
[jhielson-Alienware-15-R4:52729] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f095f09edf3]
[jhielson-Alienware-15-R4:52729] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f326f707b1b]
[jhielson-Alienware-15-R4:52733] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f095f0a6ef6]
[jhielson-Alienware-15-R4:52729] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f2444f91b1b]
[jhielson-Alienware-15-R4:52734] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe4963a7e3b]
[jhielson-Alienware-15-R4:52731] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f326f4d3df3]
[jhielson-Alienware-15-R4:52733] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f095f1f4e3b]
[jhielson-Alienware-15-R4:52729] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f326f4dbef6]
[jhielson-Alienware-15-R4:52733] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7fe496485114]
[jhielson-Alienware-15-R4:52731] [20] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f5fd320c6af]
[jhielson-Alienware-15-R4:52730] [14] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f095f2d2114]
[jhielson-Alienware-15-R4:52729] [20] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f2444d5ddf3]
[jhielson-Alienware-15-R4:52734] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f2444d65ef6]
[jhielson-Alienware-15-R4:52734] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f326f629e3b]
[jhielson-Alienware-15-R4:52733] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f326f707114]
[jhielson-Alienware-15-R4:52733] [20] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f65d9031772]
[jhielson-Alienware-15-R4:52732] [15] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f65d88fab1b]
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7fe496485830]
[jhielson-Alienware-15-R4:52731] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7fe496257a7a]
[jhielson-Alienware-15-R4:52731] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f095f2d2830]
[jhielson-Alienware-15-R4:52729] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f095f0a4a7a]
[jhielson-Alienware-15-R4:52729] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f326f707830]
[jhielson-Alienware-15-R4:52733] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f326f4d9a7a]
[jhielson-Alienware-15-R4:52733] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f2444eb3e3b]
[jhielson-Alienware-15-R4:52734] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f2444f91114]
[jhielson-Alienware-15-R4:52734] [20] [jhielson-Alienware-15-R4:52732] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f65d86c6df3]
[jhielson-Alienware-15-R4:52732] [17] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe4963a7e3b]
[jhielson-Alienware-15-R4:52731] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f095f1f4e3b]
[jhielson-Alienware-15-R4:52729] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f65d86ceef6]
[jhielson-Alienware-15-R4:52732] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7fe496485114]
[jhielson-Alienware-15-R4:52731] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f326f629e3b]
[jhielson-Alienware-15-R4:52733] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f2444f91830]
[jhielson-Alienware-15-R4:52734] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f2444d63a7a]
[jhielson-Alienware-15-R4:52734] [22] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f5fd33f2772]
[jhielson-Alienware-15-R4:52730] [15] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f095f2d2114]
[jhielson-Alienware-15-R4:52729] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f095f09ed6d]
[jhielson-Alienware-15-R4:52729] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f095f0a6ef6]
[jhielson-Alienware-15-R4:52729] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f65d881ce3b]
[jhielson-Alienware-15-R4:52732] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7fe496251d6d]
[jhielson-Alienware-15-R4:52731] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f326f707114]
[jhielson-Alienware-15-R4:52733] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f5fd2cbbb1b]
[jhielson-Alienware-15-R4:52730] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f095f0aa06b]
[jhielson-Alienware-15-R4:52729] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f095f09ed6d]
[jhielson-Alienware-15-R4:52729] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f095f0a0018]
[jhielson-Alienware-15-R4:52729] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f65d88fa114]
[jhielson-Alienware-15-R4:52732] [20] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f65d88fa830]
[jhielson-Alienware-15-R4:52732] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7fe496259ef6]
[jhielson-Alienware-15-R4:52731] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7fe49625d06b]
[jhielson-Alienware-15-R4:52731] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7fe496251d6d]
[jhielson-Alienware-15-R4:52731] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7fe496253018]
[jhielson-Alienware-15-R4:52731] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f2444eb3e3b]
[jhielson-Alienware-15-R4:52734] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f2444f91114]
[jhielson-Alienware-15-R4:52734] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f326f4d3d6d]
[jhielson-Alienware-15-R4:52733] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f326f4dbef6]
[jhielson-Alienware-15-R4:52733] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f326f4df06b]
[jhielson-Alienware-15-R4:52733] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f326f4d3d6d]
[jhielson-Alienware-15-R4:52733] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f5fd2a87df3]
[jhielson-Alienware-15-R4:52730] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f5fd2a8fef6]
[jhielson-Alienware-15-R4:52730] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f095f1f4e3b]
[jhielson-Alienware-15-R4:52729] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f65d86cca7a]
[jhielson-Alienware-15-R4:52732] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f2444d5dd6d]
[jhielson-Alienware-15-R4:52734] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f5fd2bdde3b]
[jhielson-Alienware-15-R4:52730] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f5fd2cbb114]
[jhielson-Alienware-15-R4:52730] [20] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f326f4d5018]
[jhielson-Alienware-15-R4:52733] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f326f629e3b]
[jhielson-Alienware-15-R4:52733] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f2444d65ef6]
[jhielson-Alienware-15-R4:52734] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe4963a7e3b]
[jhielson-Alienware-15-R4:52731] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f65d881ce3b]
[jhielson-Alienware-15-R4:52732] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f2444d6906b]
[jhielson-Alienware-15-R4:52734] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f65d88fa114]
[jhielson-Alienware-15-R4:52732] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f5fd2cbb830]
[jhielson-Alienware-15-R4:52730] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f2444d5dd6d]
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f65d86c6d6d]
[jhielson-Alienware-15-R4:52732] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f5fd2a8da7a]
[jhielson-Alienware-15-R4:52730] [22] [jhielson-Alienware-15-R4:52734] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f5fd2bdde3b]
[jhielson-Alienware-15-R4:52730] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f2444d5f018]
[jhielson-Alienware-15-R4:52734] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f65d86ceef6]
[jhielson-Alienware-15-R4:52732] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f5fd2cbb114]
[jhielson-Alienware-15-R4:52730] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f65d86d206b]
[jhielson-Alienware-15-R4:52732] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f2444eb3e3b]
[jhielson-Alienware-15-R4:52734] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f5fd2a87d6d]
[jhielson-Alienware-15-R4:52730] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f5fd2a8fef6]
[jhielson-Alienware-15-R4:52730] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f65d86c6d6d]
[jhielson-Alienware-15-R4:52732] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f65d86c8018]
[jhielson-Alienware-15-R4:52732] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f65d881ce3b]
[jhielson-Alienware-15-R4:52732] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f5fd2a9306b]
[jhielson-Alienware-15-R4:52730] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f5fd2a87d6d]
[jhielson-Alienware-15-R4:52730] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f5fd2a89018]
[jhielson-Alienware-15-R4:52730] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f5fd2bdde3b]
[jhielson-Alienware-15-R4:52730] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 4 with PID 0 on node jhielson-Alienware-15-R4 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:52725] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:52725] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

jhielson commented 2 years ago

Hi,

I got a similar error using a different laptop:

special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.

Can you help me to solve that? I believe we are getting close.

jhielson commented 2 years ago

Hi,

The error comes from line 1093 of the following code: https://nrn.readthedocs.io/en/latest/doxygen/nrncore__callbacks_8cpp_source.html. I am not sure how to fix that.

pramodk commented 2 years ago

@jhielson: Just FYI, we (neuron/coreneuron developers) are meeting today+tomorrow for a hackathon. In case you have time / you would like, you can join for some time (via zoom) and we can take a look at this together. Can send you the zoom link if you are interested.

jhielson commented 2 years ago

Hi @pramodk, I would appreciate that.

iomaganaris commented 2 years ago

Hello @jhielson ,

I sent you a zoom invite to your email. You can join our zoom meeting and sync there

BlueBrain / CoreNeuron

Error: undefined symbol: _Z9nrnpy_hocv #827