[Re] Hippocampal Phase-Amplitude Coupling unearthed again

[hhentschke]

This is a submission to the Ten Years Reproducibility Challenge.

Original article: Hentschke, H., Perkins, M.G., Pearce, R.A., & Banks, M.I. (2007) Muscarinic blockade weakens interaction of gamma with theta rhythms in mouse hippocampus. Eur.J.Neurosci., 26, 1642–1656.

PDF URL: https://github.com/hhentschke/ReScience_10yrReproChallenge/blob/master/paper/article.pdf Metadata URL: https://github.com/hhentschke/ReScience_10yrReproChallenge/blob/master/paper/metadata.yaml Code URL: https://github.com/hhentschke/ReScience_10yrReproChallenge

Scientific domain: Neuroscience Programming language: Matlab Suggested editor: Thomas Arildsen

[rougier]

@hhentschke Thanks for your submission and sorry for the delay.

@ThomasA @oliviaguest @benoit-girard @eroesch @gdetor Can yu handle this submission for the Ten Years Reproducibility Challenge (only 1 reviewer needed) ?

[oliviaguest]

Happy to take this on!

[oliviaguest]

👋 @TomDonoghue would you be interested in reviewing this?

[TomDonoghue]

Hey @oliviaguest - thanks for thinking of me!

The 10 years reproducibility challenge looks really interesting, and I would be happy to review this submission! Are there are specific guidelines for reviewing this kind of article / submission that I should be aware of?

[oliviaguest]

Fantastic, @TomDonoghue — thank you! I actually am not sure if there are specific ones for this type [please fill me in @khinsen @rougier @benoit-girard] but generally, see this: http://rescience.github.io/edit/

[oliviaguest]

Hey @TomDonoghue — no pressure at all — but would you be able to give a rough ETA for your review? ☺️

[TomDonoghue]

Definitely this week! Current plan should have it done by Wednesday!

[TomDonoghue]

Okay, so I have read through the original paper, the submitted ReScience paper, and looked through the repository. This paper reports on an attempt to re-run the analyses from a paper from 2007 examining theta & gamma oscillations, and cross frequency interactions, in mouse hippocampus, while manipulating muscarinic acetylcholinergic activity with atropine. I think this is a very interesting and overall good submission for the 10 years reproducibility challenge. I commend the author on picking up this very interesting challenge, and doing the work to step through old code, organize it, and try to get it all working! I have some comments, mainly addressed at hopefully trying to clarify the narrative of the paper, and a couple comments / questions about the organization of the repository.

Review Related Comments

I have a couple open questions about the review process, that perhaps @oliviaguest can help with a bit. I read the general ReScience review comments, and the notes on this challenge in particular, but was left with a couple open questions.

• Am I supposed to be able to and try to re-run the code? The general ReScience guidelines suggest this, but I'm unclear whether this is also the case for this particular challenge. Since there is no data provided, I cannot run the code (I'm not sure whether this data could be released by the author). The descriptions in the repo are organized in terms of trying to get the code to run, but this isn't something I can currently try and do.
• To what extend am I supposed to review the code? Since I can't try and run it, and I'm unclear on if, and in what way I should review it. So far I haven't worked through the code too systematically.
• Are there guidelines for how much needs to be reproduced? This submission directly reproduces 4 sub-panels (5B, 5C, 7A, 9A) from the original paper. I'm not sure if there is a recommended threshold. (I don't necessarily think the paper should require more, but I did add comments below about documenting a bit what what did get re-run, and wasn't sure if there was some external guidelines of scoping out how much should be re-run).

Paper Comments

My main comment of the paper, is that I think the paper could be a bit clearer about what it tried to reproduce, and in what ways this was / was not successful.

• For example, I think it would be easier to follow if there was a more clear & explicit note about retrieving the data, in the main text, rather than a link to the acknowledgements. Reading the paper, by the end I could piece together that some of the data was not retrieved, but I think this should be just directly stated directly and early, as we "we attempted to access the code, and found X, and for the data, were able to access Y".

• It is a bit unclear how successful the replication attempt was. In section 1.2, you say "reproducibility of the results was harder and only partly successful". I'm slightly unclear, even after reading the full paper, whether this refers specifically to not finding all of the data to re-run things completely, and/or if some things couldn't be run for other reasons. I would suggest being more explicit and separating / noting success or lack thereof due to data, and success or lack thereof due to software issues. The end of part 3 seems to allude to the notion that everything you tried to run did work in the end. If so, I think the last sentence of part 3 could clarify whether this relates to re-running all key findings, or selected key findings (from the original paper) as it's unclear how much was attempted (see next point).

• Relatedly, how much of the paper was attempted to be replicated? The paper reports and recreates 4 figure sub-panels, as a sub-selection of the original paper. It is unclear if these are selected as key elements of the original, and/or if they are the only ones able to be re-created with limited data, and/or if anything was attempted to be re-run and failed for software reasons. I'm not sure if there is a recommended threshold for how much should be attempted, but I would suggest that perhaps at least a brief description of how many figures were attempted to be re-run, and why these specific figures were included in the replication paper would be useful. One possibility might be to state more specifically what the key findings are of the original paper that you will directly reproduce here (including if this is a sub-selection). Maybe this could be something like "from the original paper, we selected key findings X, Y, Z, as demonstrated in original figures X1, X2, X3, and report on those figures here". If more was run, that is not reported in the ReScience paper, these extra figures could perhaps be added and noted in the repository?

• There are a couple parts that redirect to other sections, that I don't quite follow. In the second paragraph of part 3, it states finding scripts was partly successful "(see above)", but the above section seems to indicate you found all the scripts? I think maybe you are alluding to some difficulties finding parameter files (this also refers the reader to the acknowledgments). I think this section should be clearer on stating directly what was found, and what was not (without referencing other sections). This description could / should perhaps be consolidated and moved into section 2.

• Figure 2 is included, I believe, to demonstrate that rmouse was able to be re-run? I think this makes sense, but that the figure caption (or perhaps somewhere in text) could acknowledge and describe this slightly. As of right now, there is no description to really understand how this figure was created or what the panels show. Perhaps the main text could briefly describe something like "to investigate the re-use of the rmouse tool, X data was used to generate sample plots, briefly described as Y". This could also / instead be described in the repository, but I didn't find any reference to making this figure in the repository.

• The guidelines for the challenge recommend saying whether "someone else would be able to do the required updates". This is something you could perhaps note in your final comments in the paper.

Small / Stylistic Notes:

• there is an opened but not closed parenthesis on line 5 of page 3
• page four, bullet point labelled '1', "two dozens" should be "two dozen"
• in the figure 2 caption. I'm not sure I understand the 'working style' part - I think maybe those words could be dropped?
• figure 3, legend for D should reference that this figure is 9A from the original paper

Code Notes

I do find the repository a little hard to understand what everything is. Could the layout of the files be described a bit more?

For example:

• in scripts_paper it might be easier if Instructions_for_figures.md was called README.md, so it would render in this folder

  • The first bullet point cuts off mid sentence
  • I'm not sure I really follow these instructions. They seem to describe running things in scripts_data. It's unclear what reservoir is.

• in scripts_data, I think there needs to be some kind of description of these files. What are the folders? Files for each subject?

  • Right now I don't know how to navigate these files. Indicating which are parameter files, if and where some kind of master scripts are, and what to focus on would be helpful

• Based on the guidelines, I think the repo is supposed to have a copy of the original code & of the updated version?

  • I think the repo only contains the updated version? Does the original commit in the repo reflect the original, unedited code?
  • Perhaps note on if all of this code has been updated and if / how to see the original version would be useful

[oliviaguest]

@koustuvsinha @rougier can you shed some light on the questions above with respect to the review? Your input would be greatly appreciated.

[hhentschke]

@TomDonoghue Thank you very much for the very thorough review! I'll answer your points in detail once your review-related comments have been commented upon. Just a quick explanation of my take on the challenge. My understanding, garnered from the description of the challenge (https://rescience.github.io/ten-years/), was that a depiction of the journey was more important than the destination. Given that unsuccessful attempts at reproduction were admissable, provided the story is interesting, made me believe that a proof of concept of repeatability plus a selected choice of actual figure reproductions would be adequate. Also, compared to a few early submissions I peeked at, which were quite short (2-4 pages), my manuscript is lengthy; showing reproductions of all nine figures of the orignal paper in the manuscript would certainly be off-putting for readers. That being said, I agree that this goal should be communicated early on. Will do so.

[TomDonoghue]

Hey @hhentschke - I agree, and sorry, I didn't mean to imply that I think the paper should have to include all of the original figures! Related to that point my main suggestion would be describe a bit more explicitly the plan and sub-selection chosen for replication. I also understand the call as more of a description of the replication attempt, and did scan and see at least one other submission that explicitly talked about replicating a subset of the paper. In terms of if there is a threshold for how much should be re-run, I'll leave that to the editors to say anything more about.

[hhentschke]

Hi @TomDonoghue, I decided to move forward and work on your comments, irrespective of the issue of completeness. You were quite right with the points you mentioned, and I changed the manuscript in the following ways:

• It is now clearly stated in the main text that only a subset of the experimental data could be retrieved, but that otherwise the reproduction was successful (in 'Expectations and results in a nutshell', also also at the end of section 4 (formerly 3)). I also cut down the verbose acknowledgements.

• There are now clear statements about the exact scope of the reproduction: in the (new) Abstract, and in the last paragraph of the new section 'Code base and scope of reproduction'. The exact figure panels from the paper I had in mind for reproduction are listed in the legend of Figure 3.

• It is stated that all other figure panels featuring the horizontal bar plots could be reproduced (end of section 4).

• The multiple and partly circular redirections to the various sections of the manuscript are gone; I have tried to be as explicit as possible.

• The purpose, underlying data and details of Figure 2 are given in the caption, and the term 'working-style' is gone.

• The 'Conclusion and personal evaluation' has been extended by a comment on the feasibility of the required changes to the code by anyone else.

• The small stylistic glitches have been taken care of.

As to the code and repository, I have reorganized, corrected and extended the documentation of the repository such that hopefully the structure of the code is much clearer now. In detail:

• In scripts_paper, Instructions_for_figures.md has been renamed Readme.md, as you suggested, and the content split up and clarified
• There is a new readme for scripts_data
• Subfolder 'reservoir' is gone as it wasn't very helpful
• the basic 'master script' is explicitly mentioned

The repository does in fact contain the code in its original, unchanged version, including a commit message explaining its state. See the initial commit from April 3rd this year. Including the latest commit, there is a total of 15 commits (https://github.com/hhentschke/ReScience_10yrReproChallenge/commits/master), documenting the progression of changes.

Once again, thanks for your efforts! I'm curious to hear what the editors say.

[oliviaguest]

@TomDonoghue are you currently happy with the changes, etc.? ☺️

[TomDonoghue]

Hey @hhentschke (and cc: @oliviaguest):

First of all, I would like to apologize for the large delay in replying here. It seems I missed the original notification, and happened to notice this when scrolling through open issues, so my bad and my apologies for that.

I have revisited the project, and checked the updated repository and the updated paper. I am happy with the updates to the paper, and I find the organization of the updated version to be much easier to follow. I think the updates to the repository also help to organize and describe the files. When I re-read the paper now, I don't have any other notable comments or concerns about this paper.

In terms of my original questions to the editors - I think the updated paper is quite clear on replicating the usage of the a processing toolbox and a key set of results from the original paper, and I do not have any worries about the extent of the replication. As far as I'm aware, reviewing and/or re-running the code is not strictly required. Overall, I don't feel there is any outstanding issues that necessarily require anything more here.

All in all, I consider this paper is an interesting and useful submission for the 10 Years Challenge, as I understand the call, and in my capacity as reviewer, I would be happy to say this submission looks good to me to be accepted for that.

Thank you for the work on the updates @hhentschke - I enjoyed reading and reviewing this paper, congrats on the interesting submission!

@oliviaguest - please let me know if there is anything else you need from me for this submission!

[hhentschke]

@TomDonoghue, no problem about the delay - after all it's vacation time, and this is not a project with a deadline. Thank you very much for the time and effort you invested in the review, and your kind words!

[rougier]

@oliviaguest Gentle reminder

[oliviaguest]

@hhentschke do you have an update on the status of your submission, given the reviewer feedback? 😊

[hhentschke]

@oliviaguest I submitted a thorough revision on June 15th following the first review, which @TomDonoghue approved of about a month ago. I'm not familiar with the submission procedure - is there anything up to me right now?

[oliviaguest]

@hhentschke Yes, can you please fill in the missing details in the https://github.com/hhentschke/ReScience_10yrReproChallenge/blob/master/paper/metadata.yaml file, please? Set acceptance date as today. 🎉

This is relevant info: https://github.com/ReScience/ReScience/issues/48#issuecomment-689405707. And my ORCiD is: https://orcid.org/0000-0002-1891-0972 — let me know if anything is not clear. ☺️

[hhentschke]

Yay! Thanks a lot for your efforts! I left the last but one block ("article") in the metadata.yaml open because I'm not sure whether that final step of assigning article number, DOI and pdf url is up to you (including re-making the file) or me. Will do so if you tell me to.

[oliviaguest]

~@TomDonoghue what's your ORCID?~

No worries, found it.

[oliviaguest]

@rougier is this all done now? How do we update the website?

@hhentschke please merge https://github.com/hhentschke/ReScience_10yrReproChallenge/pull/1 😊 when you can!

[rougier]

I just updated the website and the entry is online now.

[oliviaguest]

@hhentschke yay! 🥳

Congratulations! http://rescience.github.io/bibliography/Hentschke_2020.html