10 year challenge: Davison, Feng and Brown (2003)

[apdavison]

Original article: https://doi.org/10.1152/jn.00623.2002

PDF URL: https://github.com/apdavison/bulbnet-reproduction/blob/master/reproducibility_challenge_2020/article.pdf Metadata URL: https://github.com/apdavison/bulbnet-reproduction/blob/master/reproducibility_challenge_2020/metadata.yaml Code URL: https://github.com/apdavison/bulbnet-reproduction

Scientific domain: Computational Neuroscience Programming language: Hoc Suggested editor: Tiziano Zito or Georgios Detorakis or Etienne Roesch

Note that this is paper #5 in https://github.com/ReScience/ten-years/issues/1

[rougier]

Thank your for your submission, we'll assign an editor soon.

@otizonaizit @gdetor @eroesch Can you edit this submission for the Ten Years Reproducibility Challenge (only 1 reviewer needed. Reproduction using Neuron and hoc language.

[otizonaizit]

I can edit this.

[otizonaizit]

@rossant : would you be up to review this?

[otizonaizit]

@pdebuyl : would you be up to review this?

[otizonaizit]

@pdebuyl : friendly ping. Can you review this?

[pdebuyl]

sorry @otizonaizit I am not familiar with either the domain or the language, maybe someone else? I can do it after my other ongoing reviews if necessary.

[otizonaizit]

OK @pdebuyl I'll look for another reviewer :-)

[otizonaizit]

@heplesser : do you think you could review this? It is for the Ten Years Reproducibility Challenge, with a simplified review process (only one reviewer).

[otizonaizit]

@heplesser : also, have a look at the guidelines for authors: I did not find them in the website...

[heplesser]

@otizonaizit I work rather too closely with @apdavison to be a sufficiently impartial reviewer, I'm afraid.

[otizonaizit]

It's getting difficult to get a reviewer. Sorry for the mass call, but is anyone of the Neuroscience reviewers up for review this paper? @vitay @neuronalX @piero-le-fu @benureau @damiendr @anne-urai @stephanmg @pietromarchesi @miladh @mlosch @degoldschmidt @Vahidrostami @TiinaManninen @appukuttan-shailesh @rcaze @junpenglao @thmosqueiro @cJarvers @rgutzen @schmidDan

[neuronalX]

too far from my expertise, sorry.

[cJarvers]

Same here, sorry. Not my area of expertise.

[schmidDan]

Same for me, and at the moment I'm still occupied with another review, sorry.

[appukuttan-shailesh]

I am a postdoc under Andrew Davison (one of the authors), so probably not ideal for me to take this up ;-)

[rougier]

I'll review it. See also https://github.com/apdavison/bulbnet-reproduction/issues/1

[otizonaizit]

On 27 May 2020 19:17:06 CEST, "Nicolas P. Rougier" notifications@github.com wrote:

I'll review it

Great, thanks Nicolas!

[otizonaizit]

@rougier : ping!

[rougier]

Thanks for the reminder. I'll try to do it by Friday night (I've started playing with the code). Ping me again if not done.

[rougier]

This paper was a real pleasure to review. It is well written and I had no problem to re-run most of the simulations (some were very long and I did no test them) and got the expected results with some discrepancies.

What is a bit troubling though is that for some figures, I got results very similar to the original article and a bit different from the replication (see figure 4B below: I think my re-run is closer to the original). For some other figures, my re-run is a bit farther from both the original and the replication. See for example figure 2b below. I guess this is linked with the initialization of the pseudo-random number generator but the qualitative behavior seems a bit impacted. Any idea what could be the explanation ?

Apart from that, I've only a few minor comments and I spotted a typo.

• "Compartmental models of the mitral and granule cells of the had..." -> "Compartmental models of the mitral and granule cells of olfactory bulb the had..."

• The time estimation of the different simulation could be added in the README since there is no progress indicator during a simulation and you have no idea if the simulation is running as expected.

• In the dependencies, maybe you can add gnuplot as well.

• I was lucky enough to have a warning about a failed conversion of ICC profile that indicated me where was the result. In the output of each simulation, there is a bunch of text but nothing said about where is the actual result.

Overall, the reproduction is really nice, especially given the complexity of the model and the number of results. Congratulations. And I was happy to learn that Gnuplot still exists and seem to be pretty stable.

[otizonaizit]

@apdavison : could you comment on @rougier 's review? I think we are very near to publishing the paper. It would be nice to hear your comments so that we can move forward with this :-)

[apdavison]

@otizonaizit yes, I haven't forgotten about it ;-) Based on @rougier's comments, I want to make a few tweaks to give more progress information, I'll do that this week.

[apdavison]

Many thanks @rougier for your kind comments.

I've fixed the typo, added information about dependencies and the expected simulation times to the README, and the scripts now print the location of the outputs.

The differences in the figures are certainly due to the random number generator initialisation. For individual granule cell responses (Fig 2B using the numbering from the original manuscript, Fig 1B in the reproduction), the qualitative behaviour can indeed change considerably, as the seed affects the connectivity and hence the number of inputs each cell receives. At the network level, however, the qualitative behaviour is unchanged, as reflected in the mitral cell IPSC (Fig 2C; a disynaptic response) and in the granule cell population spike time histogram (Fig 2E). To confirm this, I've added an extra panel 'F' to Figure 1 (the reproduction of the original Fig. 2) which overlays the original mitral cell IPSC trace from 2003 on top of 50 runs of the simulation with different random seeds. The original trace lies within the range of responses seen with different seeds.

By the way, on re-running all of the simulations in a fresh install, I actually get results closer to what you posted above than to the figure in the first version of the manuscript. It is possible I was experimenting with different seeds when I produced the first version of the figures.

Finally, to increase the reproducibility further, I've built the manuscript using the Sumatra LaTeX package: each figure now links to a record on my website of the simulation that produced that figure.

[rougier]

@apdavison Thanks for the explanation. Maybe you can add a small paragraphs in the manuscript with the above explanations. @otizonaizit I think we can accept the paper. There is one pending suggestion that @apdavison is free to take into account or not but apart from that, I think the paper's ready for publication.

[otizonaizit]

Thanks @rougier ! @apdavison : let me know if you want to add the mentioned paragraph. You'll get a PR from me on your repo to update the metadata before publication.

[apdavison]

Many thanks @rougier, I've added more explanation to the manuscript as you suggested.

@otizonaizit I think everything is now ready to go.

[otizonaizit]

Hi all,

as soon as this PR https://github.com/ReScience/articles/pull/14 and this PR https://github.com/ReScience/rescience.github.io/pull/93 are merged, the paper should appear on the ReScience website!

Congratulations @apdavison !

[otizonaizit]

The paper is online: https://rescience.github.io/read/

[apdavison]

Many thanks @rougier and @otizonaizit !