BlueBrain / CoreNeuron

Simulator optimized for large scale neural network simulations.
BSD 3-Clause "New" or "Revised" License
135 stars 39 forks source link

CI: add olfactory-bulb-3d testing #665

Closed alexsavulescu closed 2 years ago

alexsavulescu commented 3 years ago

For the Frontiers paper we are showcasing performance results from the olfactory-bulb-3d model. For the paper we used this branch from a custom fork that has some small changes in the code and the benchmarking setup. To make sure that the circuit is running properly in the future, it would be useful to add an test that runs a small simulation of this model. To reduce the model it's possible to select a smaller portion of the whole circuit by modifying this line and also set a smaller simulation time in the same file. Here are some examples on how to run this model.

iomaganaris commented 3 years ago

Adding this model in the test suite of NEURON takes around 30-90 seconds for each execution (NEURON, CoreNEURON direct, CoreNEURON file mode on CPU) depending how many tests are running in parallel even when setting:

runsim.build_part_model([1], [])

in https://github.com/HumanBrainProject/olfactory-bulb-3d/blob/2to3/sim/bulb3dtest.py#L19. I suspect that this is due to the high number of Mitrals generated. This is the output of the execution of bulb3dtest.py with NEURON using 2 ranks on my local machine:

105: NEURON -- VERSION 8.0a-688-g76c601a06+ magkanar/olfactory_test (76c601a06+) 2021-10-29
105: Duke, Yale, and the BlueBrain Project -- Copyright 1984-2021
105: See http://neuron.yale.edu/neuron/credits
105: 
105: Additional mechanisms from files
105:  "./ampanmda.mod" "./distrt.mod" "./fi.mod" "./fi_stdp.mod" "./gap.mod" "./Gfluct.mod" "./kamt.mod" "./kdrmt.mod" "./ks.mod" "./naxn.mod" "./orn.mod" "./ostimhelper.mod" "./ThreshDetect.mod"
105: Building part model
105: glomid : 1 gids : {5, 6, 7, 8, 9, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654}
105: determine_connections  5.519999980926514
105: net_mitral_centric after import mgrs elapsedtime 5.53
105: Building part model
105: glomid : 1 gids : {5, 6, 7, 8, 9, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654}
105: 15 mitrals created and connections to mitrals determined elapsedtime 0.15
105: Mitral 11112 and mTufted 0 cells connection infos. generated (it=2,err=0%) elapsedtime 5.18
105: rank_gconnections known elapsedtime 0.00999999
105: granule gids known on each rank elapsedtime 0
105: mk_gconnection_info (#granules = 11112) elapsedtime 0
105: mitrals registered elapsedtime 0
105: 0 blanes created elapsedtime 0
105: blanes registered elapsedtime 0
105: 11112 granules built elapsedtime 0.27
105: granules registered elapsedtime 0.0600002
105: 0 middle tufted to blanes connections generated elapsedtime 0
105:    11112 granules are generated elapsedtime 0
105: 0 blanes to granule connections generated elapsedtime 0.35
105: nmultiple =  22224
105: 22224 ThreshDetect for reciprocalsynapses constructed elapsedtime 0.71
105: 0 mt to bc elapsedtime 0
105: 0 bc to gc elapsedtime 0
105: build_net_round_robin elapsedtime 0
105: round robin setuptime  12.259999990463257
105: build_subnet_round_robin  6.730000019073486
105: Gap junctions built elapsedtime 0
105: NetCon count =  22224
105: Total # compartments =  61863
105: OdorStim Onion start=50 dur=1000 conc=1e+09
105: total setup time  12.27999997138977
105: mindelay = 1
105: t=0 wall interval 0.18
105: init time = 0
105: cvode active= False
105: weights reset not active
105: init time = 0
105: spike2file call#1 spikes=33487 sorted using 2 ranks, writing to 1 files in 0.03 sec
105: psolve did not advance time from t=49.96875 to tnext=50
105: 
105: Solver time : 15.59
105: runtime = 15.66
105: Load Balance = 0.997088
105: 
105:               comp          spk        split          gap 
105:  avg         15.52         0.00         0.00         0.06 
105:  max         15.57         0.01         0.00         0.11 
105: weight_files olfactory.weight.dat.[0:2] write time 0.0400002 s
105: vm recorded  write time 0 elapsedtime 15.92
105: total elapsed time  28.19000005722046

We need to find a way to make the generated circuit smaller so that it can run in less time so that it can be integrated in the NEURON test suite. @nrnhines @pramodk as you know this model better could you provide some help? Thank you!

pramodk commented 3 years ago

@iomaganaris : not sure how much we can reduce but I am thinking that 20-30 seconds of execution time is reasonable for such complex model. Hence, for such external/large models I am thinking if we should test only specific configurations. E.g. Questions while adding such model test are:

One can say it would be sufficient to test 1) mod2c as well as NMODL 2) in-memory mode 3) GPU build executing on CPU as well as GPU would be sufficient.

iomaganaris commented 3 years ago

As discussed offline I agree that this model doesn't need to run in every configuration. One useful configuration definitely is:

1) mod2c as well as NMODL 2) in-memory mode 3) GPU build executing on CPU as well as GPU would be sufficient.

Since it might not be very easy to make it smaller it might be useful to all a cmake option in NEURON like -DNRN_ENABLE_TEST_MODELS=traub,olfactory,dentate that enables these detailed models that run longer in only certain configurations of the CI and only if the developments wants to run the extra tests.