Open salvadord opened 4 years ago
Finally I got a chance to look back into this. I installed the lastest coreneuron version in the master version on a GCP VM with CentOS. I was able to reproduce the tut2.py with identical spikes (after fixing the randomizer issue) and I went on to try to the M1 model.
The first issue I had was an error compiling 2 the mod files (below is the error), which I fixed for now by simply removing those 2 mod files; the remaining 32 mod files compiled ok. This is the error:
x86_64/core/mod2c/catcb.cpp:32:0: warning: "_STRIDE" redefined
#define _STRIDE 1
In file included from /home/ext_salvadordura_gmail_com/CoreNeuron/include/coreneuron/sim/scopmath/newton_struct.h:4:0,
from x86_64/core/mod2c/catcb.cpp:18:
/home/ext_salvadordura_gmail_com/CoreNeuron/include/coreneuron/mechanism/mech/mod2c_core_thread.hpp:16:0: note: this is the location of the previous definition
#define _STRIDE _cntml_padded + _iml
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:34:18: error: ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
#define nrn_init _nrn_init__catcb
^
x86_64/core/mod2c/catcb.cpp:433:13: note: in expansion of macro ‘nrn_init’
static void nrn_init(NrnThread* _nt, Memb_list* _ml, int _type){
^~~~~~~~
x86_64/core/mod2c/catcb.cpp:34:18: note: previous declaration of ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
#define nrn_init _nrn_init__catcb
^
x86_64/core/mod2c/catcb.cpp:212:6: note: in expansion of macro ‘nrn_init’
void nrn_init(NrnThread*, Memb_list*, int);
^~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_init__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:442:10: error: ‘_vec_v’ was not declared in this scope
_v = _vec_v[_nd_idx];
^~~~~~
x86_64/core/mod2c/catcb.cpp:442:17: error: ‘_nd_idx’ was not declared in this scope
_v = _vec_v[_nd_idx];
^~~~~~~
x86_64/core/mod2c/catcb.cpp:443:5: error: ‘_PRCELLSTATE_V’ was not declared in this scope
_PRCELLSTATE_V
^~~~~~~~~~~~~~
x86_64/core/mod2c/catcb.cpp:74:19: error: ‘_nt_data’ was not declared in this scope
#define _ion_cai _nt_data[_ppvar[0*_STRIDE]]
^
x86_64/core/mod2c/catcb.cpp:445:9: note: in expansion of macro ‘_ion_cai’
cai = _ion_cai;
^~~~~~~~
x86_64/core/mod2c/catcb.cpp:74:19: note: suggested alternative: ‘pnt_name’
#define _ion_cai _nt_data[_ppvar[0*_STRIDE]]
^
x86_64/core/mod2c/catcb.cpp:445:9: note: in expansion of macro ‘_ion_cai’
cai = _ion_cai;
^~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:35:17: error: ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
#define nrn_cur _nrn_cur__catcb
^
x86_64/core/mod2c/catcb.cpp:454:13: note: in expansion of macro ‘nrn_cur’
static void nrn_cur(NrnThread* _nt, Memb_list* _ml, int _type){
^~~~~~~
x86_64/core/mod2c/catcb.cpp:35:17: note: previous declaration of ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
#define nrn_cur _nrn_cur__catcb
^
x86_64/core/mod2c/catcb.cpp:214:7: note: in expansion of macro ‘nrn_cur’
void nrn_cur(NrnThread*, Memb_list*, int);
^~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_cur__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:463:10: error: ‘_vec_v’ was not declared in this scope
_v = _vec_v[_nd_idx];
^~~~~~
x86_64/core/mod2c/catcb.cpp:463:17: error: ‘_nd_idx’ was not declared in this scope
_v = _vec_v[_nd_idx];
^~~~~~~
x86_64/core/mod2c/catcb.cpp:464:5: error: ‘_PRCELLSTATE_V’ was not declared in this scope
_PRCELLSTATE_V
^~~~~~~~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_jacob__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:37:19: error: ‘void coreneuron::_nrn_jacob__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
#define nrn_jacob _nrn_jacob__catcb
^
x86_64/core/mod2c/catcb.cpp:468:13: note: in expansion of macro ‘nrn_jacob’
static void nrn_jacob(NrnThread* _nt, Memb_list* _ml, int _type){
^~~~~~~~~
x86_64/core/mod2c/catcb.cpp:37:19: note: previous declaration of ‘void coreneuron::_nrn_jacob__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
#define nrn_jacob _nrn_jacob__catcb
^
x86_64/core/mod2c/catcb.cpp:215:7: note: in expansion of macro ‘nrn_jacob’
void nrn_jacob(NrnThread*, Memb_list*, int);
^~~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:38:19: error: ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
#define nrn_state _nrn_state__catcb
^
x86_64/core/mod2c/catcb.cpp:481:13: note: in expansion of macro ‘nrn_state’
static void nrn_state(NrnThread* _nt, Memb_list* _ml, int _type){
^~~~~~~~~
x86_64/core/mod2c/catcb.cpp:38:19: note: previous declaration of ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’
#define nrn_state _nrn_state__catcb
^
x86_64/core/mod2c/catcb.cpp:213:6: note: in expansion of macro ‘nrn_state’
void nrn_state(NrnThread*, Memb_list*, int);
^~~~~~~~~
x86_64/core/mod2c/catcb.cpp: In function ‘void coreneuron::_nrn_state__catcb(coreneuron::NrnThread*, coreneuron::Memb_list*, int)’:
x86_64/core/mod2c/catcb.cpp:490:10: error: ‘_vec_v’ was not declared in this scope
_v = _vec_v[_nd_idx];
^~~~~~
x86_64/core/mod2c/catcb.cpp:490:17: error: ‘_nd_idx’ was not declared in this scope
_v = _vec_v[_nd_idx];
^~~~~~~
x86_64/core/mod2c/catcb.cpp:491:5: error: ‘_PRCELLSTATE_V’ was not declared in this scope
_PRCELLSTATE_V
^~~~~~~~~~~~~~
x86_64/core/mod2c/catcb.cpp:494:1: error: expected ‘;’ before ‘{’ token
{
^
I expect those mod files being non threadsafe. I will take a look tomorrow.
Below is the output from the M1 simulation using CoreNeuron. In the original version the only 19 spikes in the model are generated by VecStims. Here the output generated by netpyne does not show those spikes; but the stats printed by coreneuron do show 29 spikes (instead of 19):
NEURON -- VERSION 7.8.0-52-gb819d6e+ master (b819d6e+) 2019-12-22
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
mod/ar_traub.mod mod/cadad.mod mod/cadyn.mod mod/cagk.mod mod/cal_mh.mod mod/cal_mig.mod mod/canin.mod mod/can_mig.mod mod/catcb.mod mod/cat_mig.mod mod/cat_traub.mod mod/h_BS.mod mod/HCN1.mod mod/h_harnett.mod mod/hin.mod mod/h_kole.mod mod/h_migliore.mod mod/ican_sidi.mod mod/IC.mod mod/IKsin.mod mod/kap_BS.mod mod/kapcb.mod mod/kapin.mod mod/kBK.mod mod/kctin.mod mod/kdmc_BS.mod mod/kdr_BS.mod mod/kdrin.mod mod/MyExp2SynBB.mod mod/MyExp2SynNMDABB.mod mod/nafx.mod mod/nap_sidi.mod mod/nax_BS.mod mod/savedist.mod mod/vecstim.mod
0
Balancing each compartment to -64 mV
Starting sim ...
Creating network of 22 cell populations on 1 hosts...
Number of cells on node 0: 151
Done; cell creation time = 1.39 s.
Making connections...
Distributing synapses based on subcellular connectivity rules...
Number of connections on node 0: 5058
Number of synaptic contacts on node 0: 22231
Done; cell connection time = 7.71 s.
Number of stims on node 0: 0
Done; cell stims creation time = 0.00 s.
Recording 2 traces of 1 types on node 0
Running simulation using CoreNEURON for 50.0 ms...
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
version id unimplemented
Additional mechanisms from files
HCN1.mod IC.mod IKsin.mod MyExp2SynBB.mod MyExp2SynNMDABB.mod ar_traub.mod cadad.mod cadyn.mod cagk.mod cal_mh.mod cal_mig.mod can_mig.mod canin.mod cat_mig.mod cat_traub.mod h_harnett.mod h_kole.mod h_migliore.mod hin.mod ican_sidi.mod kBK.mod kap_BS.mod kapcb.mod kapin.mod kctin.mod kdmc_BS.mod kdr_BS.mod kdrin.mod nafx.mod nap_sidi.mod nax_BS.mod savedist.mod vecstim.mod
Memory (MBs) : After mk_mech : Max 173.7891, Min 173.7891, Avg 173.7891
Memory (MBs) : After MPI_Init : Max 173.7891, Min 173.7891, Avg 173.7891
Memory (MBs) : Before nrn_setup : Max 173.7891, Min 173.7891, Avg 173.7891
Setup Done : 0.02 seconds
Memory (MBs) : After nrn_setup : Max 185.4375, Min 185.4375, Avg 185.4375
--spikebuf = 100000 --spkcompress = 0 --prcellgid = -1
--cell-permute = 0 --nwarp = 0 --ms-subintervals = 2
--ms-phases = 2 --multiple = 1 --extracon = 0
--seed = -1 --report-buffer-size = 4 --pattern = not set
--datpath = . --checkpoint = not set --restore = not set
--filesdat = files.dat --outpath = . --write-config = not set
--read-config = not set --report-conf = not set --tstop = 50
--dt = 0.05 --dt_io = 0.1 --voltage = -65
--celsius = 34 --forwardskip = 0 --mindelay = 10
--help = not set --threading = not set --gpu = not set
-mpi = not set --show = not set --multisend = not set
--binqueue = not set --skip-mpi-finalize = set
Start time (t) = 0
Memory (MBs) : After mk_spikevec_buffer : Max 185.4453, Min 185.4453, Avg 185.4453
WARNING: nrn_nrn_wrote_conc support on GPU need to validate!
Memory (MBs) : After nrn_finitialize : Max 185.4453, Min 185.4453, Avg 185.4453
psolve |========================================================| t: 50.00 ETA: 0h00m16s
Solver Time : 16.6167
Simulation Statistics
Number of cells: 81
Number of compartments: 11411
Number of presyns: 151
Number of input presyns: 0
Number of synapses: 22231
Number of point processes: 11876
Number of transfer (gap) targets: 0
Number of spikes: 29
Number of spikes with non negative gid-s: 29
Done; run time = 16.76 s; real-time ratio: 0.00.
Gathering data...
Done; gather time = 1.70 s.
Analyzing...
Cells: 151
Connections: 5058 (33.50 per cell)
Synaptic contacts: 22231 (147.23 per cell)
Spikes: 0 (0.00 Hz)
IT2 : 0.000 Hz
PV2 : 0.000 Hz
IT4 : 0.000 Hz
IT5A : 0.000 Hz
PV5A : 0.000 Hz
IT5B : 0.000 Hz
PT5B : 0.000 Hz
SOM5B : 0.000 Hz
PV5B : 0.000 Hz
IT6 : 0.000 Hz
CT6 : 0.000 Hz
PV6 : 0.000 Hz
TPO : 0.000 Hz
TVL : 0.000 Hz
S1 : 0.000 Hz
S2 : 0.000 Hz
cM1 : 0.000 Hz
M2 : 0.000 Hz
OC : 0.000 Hz
Simulated time: 0.1 s; 1 workers
Run time: 16.76 s
Saving output as ./M1detailed.json ...
Finished saving!
Done; saving time = 0.35 s.
Here's the output of the original (non coreneuron) version for comparison:
[ext_salvadordura_gmail_com@coreneuron-2 m1]$ python3 init.py
0
Balancing each compartment to -64 mV
Starting sim ...
Creating network of 22 cell populations on 1 hosts...
Number of cells on node 0: 151
Done; cell creation time = 1.45 s.
Making connections...
Distributing synapses based on subcellular connectivity rules...
Number of connections on node 0: 5058
Number of synaptic contacts on node 0: 22231
Done; cell connection time = 8.48 s.
Number of stims on node 0: 0
Done; cell stims creation time = 0.00 s.
Recording 2 traces of 1 types on node 0
Running simulation for 50.0 ms...
Done; run time = 19.57 s; real-time ratio: 0.00.
Gathering data...
Done; gather time = 1.68 s.
Analyzing...
Cells: 151
Connections: 5058 (33.50 per cell)
Synaptic contacts: 22231 (147.23 per cell)
Spikes: 19 (2.52 Hz)
IT2 : 0.000 Hz
PV2 : 0.000 Hz
IT4 : 0.000 Hz
IT5A : 0.000 Hz
PV5A : 0.000 Hz
IT5B : 0.000 Hz
PT5B : 0.000 Hz
SOM5B : 0.000 Hz
PV5B : 0.000 Hz
IT6 : 0.000 Hz
CT6 : 0.000 Hz
PV6 : 0.000 Hz
TPO : 4.000 Hz
TVL : 4.000 Hz
S1 : 6.000 Hz
S2 : 4.000 Hz
cM1 : 10.000 Hz
M2 : 2.000 Hz
OC : 8.000 Hz
Simulated time: 0.1 s; 1 workers
Run time: 19.57 s
Saving output as ./M1detailed.json ...
Finished saving!
Done; saving time = 0.35 s.
So seems the issue might be related to vecstim.mod file. I know we had some discussion about this, so here's a link to the file: https://github.com/Neurosim-lab/netpyne/blob/coreneuron/examples/coreneuron/m1/mod/vecstim.mod
Hello @salvadord
Just a quick update on this ticket.
First, I run the circuit compiling NEURON
with gcc 8.3.0
and CoreNEURON
with NMODL
generating the ISPC
backend code and the simulation runtimes were the following using 1 rank:
Neuron
Running simulation for 50.0 ms...
Done; run time = 15.47 s; real-time ratio: 0.00.
CoreNeuron w/ NMODL+ISPC
Solver Time : 3.96631
There is still the issue with the different spikes that we need to look at and we would also like to do some improvements in NMODL
to make it transparent.
We will update you when we fix all issues and we can also help if you want to try NMODL
and ISPC
compiler.
Ioannis
that speedup is amazing! Should I test/check something to move this forward?
Hello @salvadord ,
For now it's fine. I will work the following days on the issues I encountered and I will let you know if anything is needed 👍
The following was also communicated to Salvador but I am sharing this here for backup and future reference.
We have managed running the M1 model with CoreNEURON and I have prepared a script for you to install everything and run the model with CoreNEURON. I just want to note that during this time there were a lot of issues that this simulation helped us tackle in NEURON, CoreNEURON and NMODL related to UNITS, NMODL corner cases, USEION variables and others. Here is a script I have written which takes care of:
#!/bin/bash -l
# following are modules needed for installing everything on our supercomputer
# make sure that you have available an GCC/other compiler, MPI and the needed python packages available
# for required packages of NMODL check here: https://github.com/BlueBrain/nmodl/blob/master/INSTALL.md#prerequisites
module load unstable cmake gcc hpe-mpi python-dev py-netpyne flex bison
BASE_DIR=$(pwd)
# clone repositories
clone_repos() {
cd $BASE_DIR
git clone https://github.com/neuronsimulator/nrn.git
cd nrn
git submodule update --init
cd external/coreneuron
git checkout master
git pull
git submodule update --init
cd external/nmodl
git checkout master
git submodule update --init
cd ../../../../..
git clone https://github.com/iomaganaris/netpyne.git
cd netpyne
git checkout coreneuron_ispc
}
# install ISPC compiler
install_ispc() {
ispc_version="v1.12.0"
url_os="linux"
ispc_version_suffix="b"
url="https://github.com/ispc/ispc/releases/download/${ispc_version}/ispc-${ispc_version}${ispc_version_suffix}-${url_os}.tar.gz"
wget -O ispc.tar.gz $url
mkdir $BASE_DIR/ispc && tar -xvzf ispc.tar.gz -C $BASE_DIR/ispc --strip 1
}
# install NEURON with CoreNEURON and ISPC
install_neuron_coreneuron_ispc() {
set -e
cd $BASE_DIR/nrn
rm -rf build
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=./install -DNRN_ENABLE_CORENEURON=ON -DCORENRN_ENABLE_NMODL=ON -DCORENRN_ENABLE_ISPC=ON -DCORENRN_NMODL_FLAGS="sympy --analytic codegen --force" -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_RX3D=OFF -DCORENRN_ENABLE_OPENMP=OFF -DNRN_ENABLE_BINARY_SPECIAL=ON -DNRN_DYNAMIC_UNITS_USE_LEGACY=ON -DCORENRN_ENABLE_LEGACY_UNITS=ON -DCMAKE_ISPC_COMPILER=$BASE_DIR/ispc/bin/ispc
make -j40
make install
set +e
}
# setup environment
clone_repos
install_ispc
install_neuron_coreneuron_ispc
export PYTHONPATH=$BASE_DIR/netpyne:$PYTHONPATH
set -x
# run nrnivmodl for NEURON and CoreNEURON
cd $BASE_DIR/netpyne/examples/coreneuron/m1
$BASE_DIR/nrn/build/install/bin/nrnivmodl -coreneuron mod
# run NEURON simulation
sed -i "s#cfg.coreneuron = True#cfg.coreneuron = False#g" cfg.py
srun ./x86_64/special -mpi -python init.py
mv out_neuron.dat out_neuron_ref.dat
# run CoreNERUON simulation
sed -i "s#cfg.coreneuron = False#cfg.coreneuron = True#g" cfg.py
srun ./x86_64/special -mpi -python init.py
mv out_neuron.dat out_coreneuron_ispc.dat
echo "====== DIFF CoreNEURON w/ ISPC backend vs NEURON SPIKES ======"
sdiff -s out_coreneuron_ispc.dat out_neuron_ref.dat && echo "CoreNEURON ISPC AND NEURON SPIKES ARE SAME"
Note that I am using my own fork and branch for NetPyNE, where I have grouped the needed and working mod files with NMODL and ISPC in the mod folder. After your latest contributions to the coreneuron branch you added some file which need more handling NMODL and I have already created issues for them in NMODL. Furthermore I have made helpful additions to NetPyNE for debugging and the ability to enable the CoreNEURON simulator from NetPyNE along with other options. You should be able to run the circuit though without all of my changes, using the coreneuron branch of upstream NetPyNE, simply by removing the mod files which are still not working or need editing with NMODL. Finally some performance numbers. The circuit I used is the following:
Cells: 257 # I used 40 ranks (1 node) to run the sim, so 257*40=~10280 (10171 according to CoreNEURON)
Connections: 107774 (419.35 per cell)
Synaptic contacts: 406249 (1580.74 per cell) # 406249*40=~16249960 (16858078 according to CoreNEURON)
Spikes: 11674 (454.24 Hz). # Same as CoreNEURON w/ mod2c
Simulation time: 1500ms
The system we used consist of 1 node of 2x XEON CPUs with 40 ranks in total and 384GB RAM. For NEURON and CoreNEURON w/ mod2c we used the GCC compiler, which should be easily available on the google cloud nodes. In the results underneath you can see the simulation times of NEURON and CoreNEURON compiler with GCC and the mod2c mod file translator, as well as the simulation time of CoreNEURON while using NMODL with sympy analytic solvers and the ISPC compiler.
Solver times
Neuron: 9721.57 s
CoreNEURON w/ mod2c: 6075.75 s (~x1.6 speedup)
CoreNEURON w/ sympy & ISPC: 2038.7 s (~x4.7 speedup)
Notes on that: mod2c is similar to the code generation backend of NEURON and it generates the same spikes. Using the sympy analytic solver and the ISPC backend there are differences in the cells’ variables and spikes. The differences in spikes are partly caused by the differences in the solvers but also due to the floating point discrepancies introduced by the vectorised code produced by the ISPC compiler.
If everything is okay next step is making sure that everything is working well on GPUs. To do that it would be very helpful to be able to select a target of cells that contain all the possible mechanisms of the model, so that we can make sure that all of them work flawlessly on CPUs/GPUs.
Download model from: https://github.com/Neurosim-lab/netpyne/tree/coreneuron/examples/coreneuron/m1
Mod files compiled correctly after updating vecstim.mod (and removing unused gabab.mod)
When running simulation with coreneuron get following errors: