Closed jhielson closed 2 years ago
Hello @jhielson ,
I looked into this but I'm not entirely sure what is the issue exactly yet. Could you check whether NRN_ENABLE_PYTHON_DYNAMIC
is disabled in your NEURON
cmake build? The only way I could reproduce it is by having it enabled while the default is disabled.
I'll come back to you when I have more details about this issue
Thanks!!! I believe that might be the problem. It was off when I built the package.
-- Python | ON
-- EXE | /usr/bin/python3
-- INC | /usr/include/python3.8
-- LIB | /usr/lib/x86_64-linux-gnu/libpython3.8.so
-- MODULE | ON
-- DYNAMIC | OFF
I don't have my laptop with me right now to test it but I'll try that as soon I return to the lab. And I will let you know here. Thank you again.
Dear @iomaganaris,
Sorry, I believe I have misunderstood your message. It was disabled the whole time as you can see below:
Command:
cmake .. -DNRN_ENABLE_CORENEURON=ON -DCORENRN_ENABLE_GPU=ON -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_RX3D=OFF -DCMAKE_INSTALL_PREFIX=$HOME/install -DCMAKE_C_COMPILER=nvc -DCMAKE_CXX_COMPILER=nvc++ -DCMAKE_CUDA_COMPILER=nvcc
Output:
-- The C compiler identification is PGI 22.3.0
-- The CXX compiler identification is PGI 22.3.0
-- Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc
-- Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting build type to 'RelWithDebInfo' as none was specified.
-- 3rd party project: using Random123 from "external/Random123"
-- Found BISON: /usr/bin/bison (found version "3.5.1")
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- Found Readline: /usr/include
-- Found MPI_C: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Detected OpenMPI 3.1.5
-- -DPYTHON_EXECUTABLE not specified. Looking for `python3` in the PATH exclusively...
-- Setting PYTHON_EXECUTABLE=/usr/bin/python3
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.10", minimum required is "3.7")
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.10", minimum required is "3")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Building CoreNEURON from submodule
-- Sub-project : using coreneuron from from /home/jhielson/coreneuron_repository/nrn/external/coreneuron
-- Found HpcCodingConv: /home/jhielson/coreneuron_repository/nrn/external/coreneuron/CMake/hpc-coding-conventions
-- CORENRN_FORMATTING: OFF
-- CORENRN_TEST_FORMATTING: OFF
-- CORENRN_CLANG_FORMAT: OFF
-- CORENRN_CMAKE_FORMAT: OFF
-- CORENRN_GIT_HOOKS: OFF
-- CORENRN_GIT_COMMIT_HOOKS:
-- CORENRN_GIT_PUSH_HOOKS: courtesy-msg
-- CORENRN_STATIC_ANALYSIS: OFF
-- CORENRN_TEST_STATIC_ANALYSIS: OFF
-- Found Random123: /home/jhielson/coreneuron_repository/nrn/external/coreneuron/external/Random123
-- Found Git: /usr/bin/git (found version "2.25.1")
-- 3rd party project: using CLI11 from "external/CLI11"
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Doxygen not found, building docs has been disabled
-- Setting default CUDA architectures to 70;80
-- The CUDA compiler identification is NVIDIA 11.6.112
-- Check for working CUDA compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvcc
-- Check for working CUDA compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Found PythonInterp: /usr/bin/python3 (found version "3.8.10")
-- Found Perl: /usr/bin/perl (found version "5.30.0")
-- Found MOD2C: /home/jhielson/coreneuron_repository/nrn/external/coreneuron/external/mod2c
-- Using mod2c submodule from /home/jhielson/coreneuron_repository/nrn/external/coreneuron/external/mod2c
-- mod2c is used as APPLICATION_NAME
-- Boost found, enabling use of memory pools for Random123...
--
-- Configured CoreNEURON 8.2.0
--
-- You can now build CoreNEURON using:
-- cmake --build . --parallel 8 [--target TARGET]
-- You might want to adjust the number of parallel build jobs for your system.
-- Some non-default targets you might want to build:
-- --------------------+--------------------------------------------------------
-- Target | Description
-- --------------------+--------------------------------------------------------
-- install | Will install CoreNEURON to: /home/jhielson/install
-- docs | Build full docs. Calls targets: doxygen, sphinx
-- --------------------+--------------------------------------------------------
-- Build option | Status
-- --------------------+--------------------------------------------------------
-- CXX COMPILER | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++
-- COMPILE FLAGS | -g -O2 --c++14 -cuda -gpu=cuda11.6,lineinfo,cc70,cc80 -acc -Mautoinline -DEIGEN_DONT_VECTORIZE=1
-- Build Type | STATIC
-- MPI | ON
-- DYNAMIC | OFF
-- INC | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/include
-- OpenMP | ON
-- Use legacy units | OFF
-- NMODL | OFF
-- MOD2CPP PATH | /home/jhielson/coreneuron_repository/nrn/build/bin/mod2c_core
-- GPU Support | ON
-- CUDA |
-- Offload | OpenACC
-- Unified Memory | OFF
-- Auto Timeout | ON
-- Wrap exp() | OFF
-- SplayTree Queue | ON
-- NetReceive Buffer | ON
-- Caliper | OFF
-- Likwid | OFF
-- Unit Tests | OFF
-- Reporting | OFF
-- --------------+--------------------------------------------------------------
-- See documentation : https://github.com/BlueBrain/CoreNeuron/
-- --------------+--------------------------------------------------------------
--
Extracting link flags from target 'Threads::Threads', beware that this can be fragile. Got:
Generating link flags from path /usr/lib/x86_64-linux-gnu/libreadline.so Got: /usr/lib/x86_64-linux-gnu/libreadline.so -Wl,-rpath,/usr/lib/x86_64-linux-gnu
Generating link flags from path /usr/lib/x86_64-linux-gnu/libpython3.8.so Got: /usr/lib/x86_64-linux-gnu/libpython3.8.so -Wl,-rpath,/usr/lib/x86_64-linux-gnu
Generating link flags from path /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so Got: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so -Wl,-rpath,/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib
--
-- Configured NEURON 8.2.0
--
-- You can now build NEURON using:
-- cmake --build . --parallel 8 [--target TARGET]
-- You might want to adjust the number of parallel build jobs for your system.
-- Some non-default targets you might want to build:
-- --------------+--------------------------------------------------------------
-- Target | Description
-- --------------+--------------------------------------------------------------
-- install | Will install NEURON to: /home/jhielson/install
-- | Change the install location of NEURON using:
-- | cmake <src_path> -DCMAKE_INSTALL_PREFIX=<install_path>
-- docs | Build full docs. Calls targets: doxygen, notebooks, sphinx, notebooks-clean
-- uninstall | Removes files installed by make install (todo)
-- --------------+--------------------------------------------------------------
-- Build option | Status
-- --------------+--------------------------------------------------------------
-- C COMPILER | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc
-- CXX COMPILER | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/bin/nvc++
-- BUILD_TYPE | RelWithDebInfo (allowed: Custom;Debug;Release;RelWithDebInfo;Fast)
-- COMPILE FLAGS | -g -O2 --diag_suppress=1,47,111,128,170,174,177,180,186,301,541,550,816,941,2465
-- Shared | ON
-- Default units | modern units (2019 nist constants)
-- MPI | ON
-- DYNAMIC | OFF
-- INC | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/include
-- LIB | /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so
-- Python | ON
-- EXE | /usr/bin/python3
-- INC | /usr/include/python3.8
-- LIB | /usr/lib/x86_64-linux-gnu/libpython3.8.so
-- MODULE | ON
-- DYNAMIC | OFF
-- Readline | /usr/lib/x86_64-linux-gnu/libreadline.so
-- RX3D | OFF
-- Interviews | OFF
-- CoreNEURON | ON
-- PATH | /home/jhielson/coreneuron_repository/nrn/external/coreneuron
-- LINK FLAGS | -cuda -gpu=cuda11.6,lineinfo,cc70,cc80 -acc -rdynamic -lrt -Wl,--whole-archive -Lx86_64 -lcorenrnmech -L$(libdir) -lcoreneuron -Wl,--no-whole-archive /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so
-- Legacy Units| OFF
-- Tests | OFF
-- --------------+--------------------------------------------------------------
-- See documentation : https://www.neuron.yale.edu/neuron/
-- --------------+--------------------------------------------------------------
--
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jhielson/coreneuron_repository/nrn/build
Hello @jhielson ,
It's disabled as it should be for now. Could you try prepending the NEURON
python package installation directory to your PYTHONPATH
and trying launching the special again? You can do this by running the following before your script:
export PYTHONPATH=$HOME/install/lib/python:$PYTHONPATH
Also it could be useful to share the output of the following commands:
echo $PYTHONPATH
ldd $HOME/install/lib/python/neuron/hoc.cpython-*.so
nm $HOME/install/lib/libnrniv.so
Thank you very much
Ok, I am gonna do that and let you know here. I have been testing some previous versions of NVHPC but got the same issue.
Hi,
I exported the python path as suggested and removed the "x86_64" folder. Then, I recompiled my mods file (nrnivmodl -coreneuron .
). Now, I get the following error:
No such file or directory: '/home/jhielson/install/lib/python/neuron/.data/bin/nrnivmodl'
Command:
echo $PYTHONPATH
Output:
/home/jhielson/install/lib/python:/home/jhielson/coreneuron_repository/nrn/build/lib/python:/opt/ros/noetic/lib/python3/dist-packages
Command:
ldd $HOME/install/lib/python/neuron/hoc.cpython-*.so
Output:
linux-vdso.so.1 (0x00007ffd707d0000)
libnrniv.so => /home/jhielson/install/lib/libnrniv.so (0x00007f5cc5294000)
libatomic.so.1 => /usr/lib/x86_64-linux-gnu/libatomic.so.1 (0x00007f5cc5240000)
libnvhpcatm.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvhpcatm.so (0x00007f5cc5035000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5cc4e53000)
libnvomp.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvomp.so (0x00007f5cc4151000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5cc4149000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5cc4126000)
libnvcpumath.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvcpumath.so (0x00007f5cc3ce5000)
libnvc.so => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/compilers/lib/libnvc.so (0x00007f5cc3a84000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5cc3892000)
libgcc_s.so.1 => /../lib/libgcc_s.so.1 (0x00007f5cc3877000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5cc3726000)
libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007f5cc36d6000)
libpython3.8.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f5cc3180000)
libmpi.so.40 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so.40 (0x00007f5cc2cf8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5cc57b6000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f5cc2cc8000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f5cc2c98000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5cc2c7c000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5cc2c77000)
libopen-rte.so.40 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libopen-rte.so.40 (0x00007f5cc2912000)
libopen-pal.so.40 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libopen-pal.so.40 (0x00007f5cc23f0000)
librdmacm.so.1 => /usr/lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f5cc23d1000)
libibverbs.so.1 => /usr/lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f5cc23b2000)
libnuma.so.1 => /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/comm_libs/openmpi/openmpi-3.1.5/lib/libnuma.so.1 (0x00007f5cc21a7000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5cc219d000)
libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f5cc2178000)
libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f5cc2100000)
Command:
nm $HOME/install/lib/libnrniv.so
The output of this one was too long. So I changed the command to:
Command:
nm $HOME/install/lib/libnrniv.so | grep _Z9nrnpy_hocv
Output:
00000000003f7840 T _Z9nrnpy_hocv
Dear @iomaganaris,
I have reinstalled coreneuron and now I got a different error:
mpirun -n 6 ./x86_64/special -mpi -python completeModel-myOwnSTDP.py
--------------------------------------------------------------------------
[[40883,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:
Module: OpenFabrics (openib)
Host: jhielson-Alienware-15-R4
Another transport will be used instead, although this may result in
lower performance.
NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.2a-5-g6e4f8cb0+ master (6e4f8cb0+) 2022-06-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits
Additional mechanisms from files
"./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"
Start time: 2022-06-20 12:53:29.684249
Creating network of 674 cell populations on 6 hosts...
Number of cells on node 1: 138
Number of cells on node 2: 138
Number of cells on node 3: 138
Number of cells on node 4: 138
Number of cells on node 5: 138
Number of cells on node 0: 138
Done; cell creation time = 0.24 s.
Making connections...
Number of connections on node 5: 1040
Number of synaptic contacts on node 5: 1046
Number of connections on node 0: 1061
Number of synaptic contacts on node 0: 1069
[jhielson-Alienware-15-R4:37703] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:37703] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Number of connections on node 4: 1647
Number of synaptic contacts on node 4: 1653
Number of connections on node 1: 1682
Number of synaptic contacts on node 1: 1688
Number of connections on node 2: 1683
Number of synaptic contacts on node 2: 1689
Number of connections on node 3: 1658
Number of synaptic contacts on node 3: 1664
Done; cell connection time = 3.27 s.
Adding stims...
Number of stims on node 4: 30
Number of stims on node 1: 31
Number of stims on node 0: 30
Number of stims on node 2: 31
Number of stims on node 3: 30
Number of stims on node 5: 30
Done; cell stims creation time = 0.04 s.
exp(764.628) out of range, returning exp(700)
exp(764.628) out of range, returning exp(700)
exp(834.429) out of range, returning exp(700)
exp(808.336) out of range, returning exp(700)
exp(808.336) out of range, returning exp(700)
exp(727.905) out of range, returning exp(700)
exp(727.905) out of range, returning exp(700)
exp(796.118) out of range, returning exp(700)
exp(796.118) out of range, returning exp(700)
exp(814.896) out of range, returning exp(700)
exp(814.896) out of range, returning exp(700)
exp(834.429) out of range, returning exp(700)
Running with interval func using CoreNEURON for 300000.0 ms...
5 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
5 near line 0
5 tstop=300000.0
3 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
3 near line 0
3 tstop=300000.0
2 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
1 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
1 near line 0
1 tstop=300000.0
4 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
4 near line 0
4 tstop=300000.0
0 ./x86_64/special: NEURON model for CoreNEURON requires cvode.cache_efficient(1)
0 near line 0
0 tstop=300000.0
^
^
3 ParallelContext[5].psolve( ^
1 ParallelContext[5].psolve(1)
^
4 ParallelContext[5].psolve(1)
^
0 ParallelContext[5].psolve(1)
5 ParallelContext[5].psolve(1)
1)
2 near line 0
2 tstop=300000.0
^
2 ParallelContext[5].psolve(1)
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:37703] 5 more processes have sent help message help-mpi-api.txt / mpi-abort
Basically, it says:
NEURON model for CoreNEURON requires cvode.cache_efficient(1)
Can you help me with this new issue?
Hello @jhielson,
Nice. It looks like we're making some progress. Regarding your new issue and assuming that you're using NetPyNE
, can you make sure that in your NetPyNE
SimConfig
the variable cache_efficient
is True
?
You can enable it by setting the following in your cfg.py
:
cfg.cache_efficient = True
See also: https://github.com/suny-downstate-medical-center/netpyne/blob/development/examples/M1detailed/cfg.py
Thanks. Now, it says it can not find GPU.
--------------------------------------------------------------------------
[[38348,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:
Module: OpenFabrics (openib)
Host: jhielson-Alienware-15-R4
Another transport will be used instead, although this may result in
lower performance.
NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.2a-5-g6e4f8cb0+ master (6e4f8cb0+) 2022-06-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits
Additional mechanisms from files
"./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"
Start time: 2022-06-20 13:01:36.929230
Creating network of 149 cell populations on 6 hosts...
Number of cells on node 1: 51
Number of cells on node 2: 51
Number of cells on node 3: 50
Number of cells on node 4: 50
Number of cells on node 5: 50
Number of cells on node 0: 51
Done; cell creation time = 0.05 s.
Making connections...
Number of connections on node 3: 452
Number of synaptic contacts on node 3: 460
Number of connections on node 1: 510
Number of synaptic contacts on node 1: 516
Number of connections on node 4: 542
Number of synaptic contacts on node 4: 548
Number of connections on node 0: 523
Number of synaptic contacts on node 0: 529
Number of connections on node 2: 430
Number of synaptic contacts on node 2: 436
Number of connections on node 5: 541
Number of synaptic contacts on node 5: 547
Done; cell connection time = 0.59 s.
Adding stims...
Number of stims on node 0: 30
Number of stims on node 4: 31
Number of stims on node 5: 31
Number of stims on node 1: 30
Number of stims on node 2: 30
Number of stims on node 3: 30
Done; cell stims creation time = 0.02 s.
exp(751.718) out of range, returning exp(700)
exp(751.718) out of range, returning exp(700)
exp(823.67) out of range, returning exp(700)
exp(823.67) out of range, returning exp(700)
exp(874.045) out of range, returning exp(700)
exp(874.045) out of range, returning exp(700)
exp(720.81) out of range, returning exp(700)
exp(720.81) out of range, returning exp(700)
exp(732.031) out of range, returning exp(700)
exp(732.031) out of range, returning exp(700)
exp(823.791) out of range, returning exp(700)
exp(823.791) out of range, returning exp(700)
Running with interval func using CoreNEURON for 20000.0 ms...
num_mpi=6
ERROR : Enabled GPU execution but couldn't find NVIDIA GPU!
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode -1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:39224] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:39224] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[jhielson-Alienware-15-R4:39224] 5 more processes have sent help message help-mpi-api.txt / mpi-abort
But I have one working fine.
Command:
nvidia-smi
Output:
Mon Jun 20 13:03:27 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| N/A 71C P0 47W / N/A | 1418MiB / 8192MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1136 G /usr/lib/xorg/Xorg 150MiB |
| 0 N/A N/A 1717 G /usr/lib/xorg/Xorg 583MiB |
| 0 N/A N/A 1847 G /usr/bin/gnome-shell 162MiB |
| 0 N/A N/A 2557 G /usr/lib/firefox/firefox 284MiB |
| 0 N/A N/A 14545 G ...RendererForSitePerProcess 222MiB |
+-----------------------------------------------------------------------------+
Hello @jhielson,
What is the GPU model you're trying to execute CoreNEURON on? It might be that CoreNEURON is not built for your GPU architecture and this is why CoreNEURON cannot find it. If you find the Compute Capability
for your GPU you can build CoreNEURON for it using the following CMake variable:
-DCMAKE_CUDA_ARCHITECTURES=61
for Compute Architecture
6.1
for example.
Thanks. I have the following device:
description: VGA compatible controller
product: GP104BM [GeForce GTX 1080 Mobile]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:163 memory:9c000000-9cffffff memory:70000000-7fffffff memory:80000000-81ffffff ioport:5000(size=128) memory:c0000-dffff
The compute capability seems to be 6.1.
I am going to compile coreneuron again and I will keep you informed.
Dear @iomaganaris,
Now, it is recognizing the GPU but there is a new error: Command:
mpirun -n 6 ./x86_64/special -mpi -python completeModel-100GC.py
Output:
--------------------------------------------------------------------------
[[49409,1],2]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:
Module: OpenFabrics (openib)
Host: jhielson-Alienware-15-R4
Another transport will be used instead, although this may result in
lower performance.
NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
numprocs=6
NEURON -- VERSION 8.2a-5-g6e4f8cb0+ master (6e4f8cb0+) 2022-06-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
See http://neuron.yale.edu/neuron/credits
Additional mechanisms from files
"./fi_stdp.mod" "./GP.mod" "./Izhi2003b.mod" "./izhi2007a.mod" "./izhi2007bS.mod" "./mySTDP.mod" "./spikeout.mod" "./stdwa_songabbott.mod" "./STN2.mod" "./Str.mod" "./SynExp2NMDA.mod" "./thalamus.mod"
Start time: 2022-06-20 14:42:52.738020
Creating network of 149 cell populations on 6 hosts...
Number of cells on node 1: 51
Number of cells on node 2: 51
Number of cells on node 3: 50
Number of cells on node 4: 50
Number of cells on node 5: 50
Number of cells on node 0: 51
Done; cell creation time = 0.08 s.
Making connections...
Number of connections on node 2: 428
Number of synaptic contacts on node 2: 434
Number of connections on node 5: 545
Number of synaptic contacts on node 5: 551
Number of connections on node 3: 454
Number of synaptic contacts on node 3: 462
Number of connections on node 0: 525
Number of synaptic contacts on node 0: 531
Number of connections on node 4: 542
Number of synaptic contacts on node 4: 548
Number of connections on node 1: 506
Number of synaptic contacts on node 1: 512
Done; cell connection time = 0.54 s.
Adding stims...
Number of stims on node 4: 31
Number of stims on node 2: 30
Number of stims on node 5: 31
Number of stims on node 3: 30
Number of stims on node 0: 30
Number of stims on node 1: 30
Done; cell stims creation time = 0.02 s.
exp(760.937) out of range, returning exp(700)
exp(760.937) out of range, returning exp(700)
exp(930.274) out of range, returning exp(700)
exp(930.274) out of range, returning exp(700)
exp(762.501) out of range, returning exp(700)
exp(762.501) out of range, returning exp(700)
exp(810.312) out of range, returning exp(700)
exp(810.312) out of range, returning exp(700)
exp(871.081) out of range, returning exp(700)
exp(871.081) out of range, returning exp(700)
exp(829.235) out of range, returning exp(700)
exp(829.235) out of range, returning exp(700)
Running with interval func using CoreNEURON for 20000.0 ms...
num_mpi=6
Info : 1 GPUs shared by 6 ranks per node
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
Version : 8.2.0 6215225f (2022-06-08 10:09:55 +0200)
Additional mechanisms from files
GP.mod Izhi2003b.mod STN2.mod Str.mod SynExp2NMDA.mod exp2syn.mod expsyn.mod fi_stdp.mod hh.mod izhi2007a.mod izhi2007bS.mod mySTDP.mod netstim.mod passive.mod pattern.mod spikeout.mod stdwa_songabbott.mod stim.mod svclmp.mod thalamus.mod
Memory (MBs) : After mk_mech : Max 398.7812, Min 398.0742, Avg 398.4473
GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
Memory (MBs) : After MPI_Init : Max 398.7812, Min 398.0742, Avg 398.4473
GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
Memory (MBs) : Before nrn_setup : Max 399.2930, Min 398.5078, Avg 398.8932
GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
best_balance=0.978723 ncell=47 ntype=1 nwarp=47
Setup Done : 0.00 seconds
Model size : 766.18 kB
Memory (MBs) : After nrn_setup : Max 399.2930, Min 398.5078, Avg 398.8932
GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
GENERAL PARAMETERS
--mpi=true
--mpi-lib=
--gpu=true
--dt=0.1
--tstop=1
GPU
--nwarp=65536
--cell-permute=2
--cuda-interface=false
INPUT PARAMETERS
--voltage=1000
--seed=-1
--datpath=.
--filesdat=files.dat
--pattern=
--report-conf=
--restore=
PARALLEL COMPUTATION PARAMETERS
--threading=false
--skip_mpi_finalize=true
SPIKE EXCHANGE
--ms_phases=2
--ms_subintervals=2
--multisend=false
--spk_compress=0
--binqueue=false
CONFIGURATION
--spikebuf=100000
--prcellgid=-1
--forwardskip=0
--celsius=6.3
--mindelay=1
--report-buffer-size=4
OUTPUT PARAMETERS
--dt_io=0.1
--outpath=.
--checkpoint=
Start time (t) = 0
Memory (MBs) : After mk_spikevec_buffer : Max 399.2930, Min 398.5078, Avg 398.8932
GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
Memory (MBs) : After nrn_finitialize : Max 399.9531, Min 399.1055, Avg 399.5358
GPU Memory (MiBs) : Used = 2132.000000, Free = 5953.937500, Total = 8085.937500
psolve |=========================================================| t: 1.00 ETA: 0h00m01s
Solver Time : 0.100294
Simulation Statistics
Number of cells: 282
Number of compartments: 846
Number of presyns: 481
Number of input presyns: 1137
Number of synapses: 3058
Number of point processes: 3419
Number of transfer sources: 0
Number of transfer targets: 0
Number of spikes: 12
Number of spikes with non negative gid-s: 12
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
[jhielson-Alienware-15-R4:52731] *** Process received signal ***
[jhielson-Alienware-15-R4:52731] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52731] Signal code: (-6)
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
[jhielson-Alienware-15-R4:52733] *** Process received signal ***
[jhielson-Alienware-15-R4:52733] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52733] Signal code: (-6)
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
[jhielson-Alienware-15-R4:52730] *** Process received signal ***
[jhielson-Alienware-15-R4:52730] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52730] Signal code: (-6)
[jhielson-Alienware-15-R4:52734] *** Process received signal ***
[jhielson-Alienware-15-R4:52734] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52734] Signal code: (-6)
[jhielson-Alienware-15-R4:52732] *** Process received signal ***
[jhielson-Alienware-15-R4:52732] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52732] Signal code: (-6)
[jhielson-Alienware-15-R4:52729] *** Process received signal ***
[jhielson-Alienware-15-R4:52729] Signal: Aborted (6)
[jhielson-Alienware-15-R4:52729] Signal code: (-6)
[jhielson-Alienware-15-R4:52731] [ 0] [jhielson-Alienware-15-R4:52734] [ 0] [jhielson-Alienware-15-R4:52733] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fe493b6a420]
[jhielson-Alienware-15-R4:52731] [ 1] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f326cdec420]
[jhielson-Alienware-15-R4:52733] [ 1] [jhielson-Alienware-15-R4:52729] [ 0] [jhielson-Alienware-15-R4:52732] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f65d5fdf420]
[jhielson-Alienware-15-R4:52732] [ 1] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f2442676420]
[jhielson-Alienware-15-R4:52734] [ 1] [jhielson-Alienware-15-R4:52730] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f5fd03a0420]
[jhielson-Alienware-15-R4:52730] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fe49330500b]
[jhielson-Alienware-15-R4:52731] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fe4932e4859]
[jhielson-Alienware-15-R4:52731] [ 3] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f326c58700b]
[jhielson-Alienware-15-R4:52733] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f326c566859]
[jhielson-Alienware-15-R4:52733] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f095c9b7420]
[jhielson-Alienware-15-R4:52729] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f095c15200b]
[jhielson-Alienware-15-R4:52729] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f65d577a00b]
[jhielson-Alienware-15-R4:52732] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f2441e1100b]
[jhielson-Alienware-15-R4:52734] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f2441df0859]
[jhielson-Alienware-15-R4:52734] [ 3] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f5fcfb3b00b]
[jhielson-Alienware-15-R4:52730] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7fe4932e4729]
[jhielson-Alienware-15-R4:52731] [ 4] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f2441df0729]
[jhielson-Alienware-15-R4:52734] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f2441e01fd6]
[jhielson-Alienware-15-R4:52734] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f5fcfb1a859]
[jhielson-Alienware-15-R4:52730] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f5fcfb1a729]
[jhielson-Alienware-15-R4:52730] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f5fcfb2bfd6]
[jhielson-Alienware-15-R4:52730] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7fe4932f5fd6]
[jhielson-Alienware-15-R4:52731] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f095c131859]
[jhielson-Alienware-15-R4:52729] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f095c131729]
[jhielson-Alienware-15-R4:52729] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f095c142fd6]
[jhielson-Alienware-15-R4:52729] [ 5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f65d5759859]
[jhielson-Alienware-15-R4:52732] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f65d5759729]
[jhielson-Alienware-15-R4:52732] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f65d576afd6]
[jhielson-Alienware-15-R4:52732] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f326c566729]
[jhielson-Alienware-15-R4:52733] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f326c577fd6]
[jhielson-Alienware-15-R4:52733] [ 5] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7fe4969b2e9e]
[jhielson-Alienware-15-R4:52731] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f24454bee9e]
[jhielson-Alienware-15-R4:52734] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f326fc34e9e]
[jhielson-Alienware-15-R4:52733] [ 6] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52731] [ 7] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f095f7ffe9e]
[jhielson-Alienware-15-R4:52729] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f65d8e27e9e]
[jhielson-Alienware-15-R4:52732] [ 6] /home/jhielson/install/lib/libnrniv.so(+0x1e5e9e)[0x7f5fd31e8e9e]
[jhielson-Alienware-15-R4:52730] [ 6] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52733] [ 7] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52734] [ 7] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52731] [ 8] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52729] [ 7] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52730] [ 7] ./x86_64/special(_ZN10coreneuron20core2nrn_data_returnEv+0x12cc)[0x53cbcc]
[jhielson-Alienware-15-R4:52732] [ 7] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52734] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52731] [ 9] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52733] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52734] [ 9] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52729] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52733] [ 9] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52730] [ 8] ./x86_64/special(run_solve_core+0xa5f)[0x4f639f]
[jhielson-Alienware-15-R4:52732] [ 8] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52729] [ 9] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52730] [ 9] ./x86_64/special(corenrn_embedded_run+0x8e)[0x4555ce]
[jhielson-Alienware-15-R4:52732] [ 9] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7fe4969ad929]
[jhielson-Alienware-15-R4:52731] [10] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f24454b9929]
[jhielson-Alienware-15-R4:52734] [10] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f326fc2f929]
[jhielson-Alienware-15-R4:52733] [10] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f095f7fa929]
[jhielson-Alienware-15-R4:52729] [10] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7fe4969d2c91]
[jhielson-Alienware-15-R4:52731] [11] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f65d8e22929]
[jhielson-Alienware-15-R4:52732] [10] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f24454dec91]
[jhielson-Alienware-15-R4:52734] [11] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f326fc54c91]
[jhielson-Alienware-15-R4:52733] [11] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f095f81fc91]
[jhielson-Alienware-15-R4:52729] [11] /home/jhielson/install/lib/libnrniv.so(_Z14nrncore_psolvedi+0x229)[0x7f5fd31e3929]
[jhielson-Alienware-15-R4:52730] [10] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f65d8e47c91]
[jhielson-Alienware-15-R4:52732] [11] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7fe496a6738e]
[jhielson-Alienware-15-R4:52731] [12] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f326fce938e]
[jhielson-Alienware-15-R4:52733] [12] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f244557338e]
[jhielson-Alienware-15-R4:52734] [12] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f095f8b438e]
[jhielson-Alienware-15-R4:52729] [12] /home/jhielson/install/lib/libnrniv.so(+0x205c91)[0x7f5fd3208c91]
[jhielson-Alienware-15-R4:52730] [11] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7fe496bbc0bb]
[jhielson-Alienware-15-R4:52731] [13] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f65d8edc38e]
[jhielson-Alienware-15-R4:52732] [12] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f095fa090bb]
[jhielson-Alienware-15-R4:52729] [13] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f326fe3e0bb]
[jhielson-Alienware-15-R4:52733] [13] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f24456c80bb]
[jhielson-Alienware-15-R4:52734] [13] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7fe4969d66af]
[jhielson-Alienware-15-R4:52731] [14] /home/jhielson/install/lib/libnrniv.so(_Z20hoc_object_componentv+0xc4e)[0x7f5fd329d38e]
[jhielson-Alienware-15-R4:52730] [12] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f65d90310bb]
[jhielson-Alienware-15-R4:52732] [13] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f095f8236af]
[jhielson-Alienware-15-R4:52729] [14] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f326fc586af]
[jhielson-Alienware-15-R4:52733] [14] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f24454e26af]
[jhielson-Alienware-15-R4:52734] [14] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7fe496bbc772]
[jhielson-Alienware-15-R4:52731] [15] /home/jhielson/install/lib/libnrniv.so(+0x3ef0bb)[0x7f5fd33f20bb]
[jhielson-Alienware-15-R4:52730] [13] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7fe496485b1b]
[jhielson-Alienware-15-R4:52731] [16] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f095fa09772]
[jhielson-Alienware-15-R4:52729] [15] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f65d8e4b6af]
[jhielson-Alienware-15-R4:52732] [14] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7fe496251df3]
[jhielson-Alienware-15-R4:52731] [17] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f326fe3e772]
[jhielson-Alienware-15-R4:52733] [15] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f24456c8772]
[jhielson-Alienware-15-R4:52734] [15] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7fe496259ef6]
[jhielson-Alienware-15-R4:52731] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f095f2d2b1b]
[jhielson-Alienware-15-R4:52729] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f095f09edf3]
[jhielson-Alienware-15-R4:52729] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f326f707b1b]
[jhielson-Alienware-15-R4:52733] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f095f0a6ef6]
[jhielson-Alienware-15-R4:52729] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f2444f91b1b]
[jhielson-Alienware-15-R4:52734] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe4963a7e3b]
[jhielson-Alienware-15-R4:52731] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f326f4d3df3]
[jhielson-Alienware-15-R4:52733] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f095f1f4e3b]
[jhielson-Alienware-15-R4:52729] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f326f4dbef6]
[jhielson-Alienware-15-R4:52733] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7fe496485114]
[jhielson-Alienware-15-R4:52731] [20] /home/jhielson/install/lib/libnrniv.so(_ZN6OcJump7fpycallEPFPvS0_S0_ES0_S0_+0x22f)[0x7f5fd320c6af]
[jhielson-Alienware-15-R4:52730] [14] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f095f2d2114]
[jhielson-Alienware-15-R4:52729] [20] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f2444d5ddf3]
[jhielson-Alienware-15-R4:52734] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f2444d65ef6]
[jhielson-Alienware-15-R4:52734] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f326f629e3b]
[jhielson-Alienware-15-R4:52733] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f326f707114]
[jhielson-Alienware-15-R4:52733] [20] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f65d9031772]
[jhielson-Alienware-15-R4:52732] [15] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f65d88fab1b]
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7fe496485830]
[jhielson-Alienware-15-R4:52731] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7fe496257a7a]
[jhielson-Alienware-15-R4:52731] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f095f2d2830]
[jhielson-Alienware-15-R4:52729] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f095f0a4a7a]
[jhielson-Alienware-15-R4:52729] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f326f707830]
[jhielson-Alienware-15-R4:52733] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f326f4d9a7a]
[jhielson-Alienware-15-R4:52733] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f2444eb3e3b]
[jhielson-Alienware-15-R4:52734] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f2444f91114]
[jhielson-Alienware-15-R4:52734] [20] [jhielson-Alienware-15-R4:52732] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f65d86c6df3]
[jhielson-Alienware-15-R4:52732] [17] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe4963a7e3b]
[jhielson-Alienware-15-R4:52731] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f095f1f4e3b]
[jhielson-Alienware-15-R4:52729] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f65d86ceef6]
[jhielson-Alienware-15-R4:52732] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7fe496485114]
[jhielson-Alienware-15-R4:52731] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f326f629e3b]
[jhielson-Alienware-15-R4:52733] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f2444f91830]
[jhielson-Alienware-15-R4:52734] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f2444d63a7a]
[jhielson-Alienware-15-R4:52734] [22] /home/jhielson/install/lib/libnrniv.so(+0x3ef772)[0x7f5fd33f2772]
[jhielson-Alienware-15-R4:52730] [15] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f095f2d2114]
[jhielson-Alienware-15-R4:52729] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f095f09ed6d]
[jhielson-Alienware-15-R4:52729] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f095f0a6ef6]
[jhielson-Alienware-15-R4:52729] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f65d881ce3b]
[jhielson-Alienware-15-R4:52732] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7fe496251d6d]
[jhielson-Alienware-15-R4:52731] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f326f707114]
[jhielson-Alienware-15-R4:52733] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyObject_MakeTpCall+0xab)[0x7f5fd2cbbb1b]
[jhielson-Alienware-15-R4:52730] [16] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f095f0aa06b]
[jhielson-Alienware-15-R4:52729] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f095f09ed6d]
[jhielson-Alienware-15-R4:52729] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f095f0a0018]
[jhielson-Alienware-15-R4:52729] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f65d88fa114]
[jhielson-Alienware-15-R4:52732] [20] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f65d88fa830]
[jhielson-Alienware-15-R4:52732] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7fe496259ef6]
[jhielson-Alienware-15-R4:52731] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7fe49625d06b]
[jhielson-Alienware-15-R4:52731] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7fe496251d6d]
[jhielson-Alienware-15-R4:52731] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7fe496253018]
[jhielson-Alienware-15-R4:52731] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f2444eb3e3b]
[jhielson-Alienware-15-R4:52734] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f2444f91114]
[jhielson-Alienware-15-R4:52734] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f326f4d3d6d]
[jhielson-Alienware-15-R4:52733] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f326f4dbef6]
[jhielson-Alienware-15-R4:52733] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f326f4df06b]
[jhielson-Alienware-15-R4:52733] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f326f4d3d6d]
[jhielson-Alienware-15-R4:52733] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74df3)[0x7f5fd2a87df3]
[jhielson-Alienware-15-R4:52730] [17] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f5fd2a8fef6]
[jhielson-Alienware-15-R4:52730] [18] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f095f1f4e3b]
[jhielson-Alienware-15-R4:52729] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f65d86cca7a]
[jhielson-Alienware-15-R4:52732] [22] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f2444d5dd6d]
[jhielson-Alienware-15-R4:52734] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f5fd2bdde3b]
[jhielson-Alienware-15-R4:52730] [19] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f5fd2cbb114]
[jhielson-Alienware-15-R4:52730] [20] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f326f4d5018]
[jhielson-Alienware-15-R4:52733] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f326f629e3b]
[jhielson-Alienware-15-R4:52733] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f2444d65ef6]
[jhielson-Alienware-15-R4:52734] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe4963a7e3b]
[jhielson-Alienware-15-R4:52731] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f65d881ce3b]
[jhielson-Alienware-15-R4:52732] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f2444d6906b]
[jhielson-Alienware-15-R4:52734] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f65d88fa114]
[jhielson-Alienware-15-R4:52732] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7f5fd2cbb830]
[jhielson-Alienware-15-R4:52730] [21] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f2444d5dd6d]
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f65d86c6d6d]
[jhielson-Alienware-15-R4:52732] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x590a)[0x7f5fd2a8da7a]
[jhielson-Alienware-15-R4:52730] [22] [jhielson-Alienware-15-R4:52734] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f5fd2bdde3b]
[jhielson-Alienware-15-R4:52730] [23] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f2444d5f018]
[jhielson-Alienware-15-R4:52734] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f65d86ceef6]
[jhielson-Alienware-15-R4:52732] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7f5fd2cbb114]
[jhielson-Alienware-15-R4:52730] [24] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f65d86d206b]
[jhielson-Alienware-15-R4:52732] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f2444eb3e3b]
[jhielson-Alienware-15-R4:52734] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f5fd2a87d6d]
[jhielson-Alienware-15-R4:52730] [25] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7f5fd2a8fef6]
[jhielson-Alienware-15-R4:52730] [26] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f65d86c6d6d]
[jhielson-Alienware-15-R4:52732] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f65d86c8018]
[jhielson-Alienware-15-R4:52732] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f65d881ce3b]
[jhielson-Alienware-15-R4:52732] *** End of error message ***
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7f5fd2a9306b]
[jhielson-Alienware-15-R4:52730] [27] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7f5fd2a87d6d]
[jhielson-Alienware-15-R4:52730] [28] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xea8)[0x7f5fd2a89018]
[jhielson-Alienware-15-R4:52730] [29] /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7f5fd2bdde3b]
[jhielson-Alienware-15-R4:52730] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 4 with PID 0 on node jhielson-Alienware-15-R4 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
[jhielson-Alienware-15-R4:52725] 5 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[jhielson-Alienware-15-R4:52725] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Hi,
I got a similar error using a different laptop:
special: /home/jhielson/coreneuron_repository/nrn/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp:1093: void core2nrn_SelfEvent_event(int, double, int, int, double, unsigned long, int): Assertion `nc->target_ == pnt' failed.
Can you help me to solve that? I believe we are getting close.
Hi,
The error comes from line 1093 of the following code: https://nrn.readthedocs.io/en/latest/doxygen/nrncore__callbacks_8cpp_source.html. I am not sure how to fix that.
@jhielson: Just FYI, we (neuron/coreneuron developers) are meeting today+tomorrow for a hackathon. In case you have time / you would like, you can join for some time (via zoom) and we can take a look at this together. Can send you the zoom link if you are interested.
Hi @pramodk, I would appreciate that.
Hello @jhielson ,
I sent you a zoom invite to your email. You can join our zoom meeting and sync there
Hi,
I have successfully compiled and built Coreneuron on my laptop. I have also managed to compile my mod files (
nrnivmodl -coreneuron .
).However, I get the following error (
undefined symbol: _Z9nrnpy_hocv
) every time I try to run my network. Can you help me?Command
Output: