Reproduction of a comparison between operant and classical conditioning of identical stimuli in tethered Drosophila

[brembs]

Original article: Brembs B and Heisenberg M (2000): The Operant and the Classical in Conditioned Orientation of Drosophila melanogaster at the Flight Simulator. Learn Mem. 7(2): 104–115. doi: 10.1101/lm.7.2.104 PDF URL original article: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311324/pdf/x2.pdf PDF URL submitted article: https://github.com/brembs/DFS_reproduction/blob/master/article.pdf Metadata URL: https://github.com/brembs/DFS_reproduction/blob/master/DFSreproduction.yaml Code URL:
https://github.com/brembs/DFS_reproduction/tree/master/evaluation_code

Scientific domain:
Neurobiology Programming language:
Turbo Pascal, R, C++ Suggested editor:

[brembs]

I'm still in the process of getting DOIs for the repository, I hope they can be provided later?

[khinsen]

@brembs For the review, we don't need a code DOI, that can wait for later. And it's actually easier today to get a Software Heritage ID for a GitHub repository. However, the Metadata and Code URLs you cite don't work!

Update: I discovered that my GitHub superpowers allowed me to fix the two links myself. I hope you don't mind!

[brembs]

Ok, not sure where the issue was. I changed one of your links and all links work for me now. Thanks a lot! Will go and get a Software heritage ID!

[rougier]

@brembs Thanks for the submission. @gdetor @eroesch @oliviaguest @benoit-girard Can one of you edit this submission in neuroscience for the the Ten Years Reproducibility Challenge (only one reviewer needed)?

[gdetor]

@rougier I can handle this.

[rougier]

@gdetor Great, thank you.

[gdetor]

Hi @eroesch, could you review this work?

[rougier]

Gentle reminder

[gdetor]

@apdavison @benoit-girard Could one of you review this submission? Thank you.

[benoit-girard]

What would be the delay to do the review? I am really sorry, with the lockdown & the kids at home, I have very little time to allocate to additional work right now, so I can do it, but if you give me enough time...

[gdetor]

It' works for me. @rougier Is a potential delay in the review process acceptable?

[rougier]

Given the situation, I think we can have a delay yes (like my late answer). Can we target end of June? @brembs Would that be acceptable ?

[brembs]

I'm in no hurry and I have had own kid at home myself and have been late on reviews. So no pressure at all from my end.

[gdetor]

Ok, so @benoit-girard is being assigned as reviewer for this submission. Thank you.

[benoit-girard]

I'm a bit lost, I'll need some help/clarification: the PDF URL refers to the original work, not to the pdf of the replication paper; this pdf will of course be useful, but where is the pdf of the replication paper, the ReScience one?

[brembs]

Sorry, this must be my misunderstanding of the instructions! The submission is in the repository, named "article.pdf": https://github.com/brembs/DFS_reproduction That was my understanding of the instructions, sorry if I misunderstood.

[brembs]

I have updated my submission above with the direct link.

[rougier]

You updated header is actually better than the template. Maybe we should adopt it as the default.

[benoit-girard]

The author documents the reproduction of data analyses using C++ code developed in the 90s.

A few point need to be clarified:

• Could the author add a reference section to his manuscript, and provide there the exact reference to the paper that is reproduced. Otherwise, consulting only the pdf of the paper, one cannot identify the original paper with certainty.

• It seems that Fig. 1 is reproduced from the original publication. If so, please state it explicitly in the legend "(reprinted from XXX)", get the authorization from he original publisher to reproduce it here, and add the mention that the publisher will probably require.

• It is mentioned page 2 that Reinhard Wolf is a co-author of the original publication, which does not appear to be the case (the github submission refers to: Brembs B and Heisenberg M (2000)). Can the author clarify that point?

• The original paper contains many results figures (from fig. 2 to fig. 8 of the original paper). The reproduction of fig. 2 only is presented. What prevented the reproduction of the other figures? The conclusion mentions that: "However, for evaluations other than performance indices (and some other evaluations provided by the C++ application), there is currently no solution." Could the author be more precise: which evaluations are completely lost, because they were not computed by the C++ application? What prevents the author to present those that are provided by the C++ application, as done in the Fig. 2 of the present paper?

• From a formal point of view, the paper is not following the ReScience template. @rougier is it mandatory for this special issue? If yes, some work has to be done to convert the document.

[brembs]

Excellent, comments, thanks! Am teaching a course right now, but will get to it starting two weeks from now!

[gdetor]

@benoit-girard Thank you for the review. @brembs Ok, thank you for the update.

[brembs]

I have started going over the manuscript and addressed most of the issues. WRT the other figures in the original paper:

• there are three additional figures (2 panels and one full figure) that I could immediately reproduce with the identical workflow as the one I used for figure 2 (I chose this figure because of the scientific content, which is important for my research and I still use this figure in teaching).
• There is one figure, Fig. 7, which I should be able to reproduce by writing another R-script that collects the relevant PIs from the 12 CSV files and plots them. Would be tedious, but the same principle as the other figures.
• For the remaining three figures (two panels and one full figure), one could try to run the old TP code on FreePascal. It may require some tweaking of the code, but my experience with FP (many years ago) was quite decent.

Thus, with quite some additional work, one could likely get nearly all of the data reproduced, but especially the last two options require quite some time. I have described all three options in the discussion. If I should actually perform any of them, I would likely need until October or so to finish it, given my current schedule and depending on how many of them I should strive to reproduce.

[gdetor]

@brembs Thank you for the update. @benoit-girard Could you please comment on the author's response? @rougier Is there any problem with the timeline proposed by the author?

[brembs]

Just to be clear: I have explained the issues with the other figures in the text and the text is ready. I only need some more time if I really should attempt reproducing the other figures. Without the additional figures, I can resubmit any time.

[rougier]

@benoit-girard @brembs Yes, it is kind f mandatory to use the rescience template but this can be done only once accepted. @gdetor For the timeline, there's no problem but if @benoit-girard is ok with the not having the supplementary figures, I think we're good to go if @brembs also agree.

[brembs]

I'll do whatever the reviewers say :-) WRT to the template: what's a good pipeline from GDocs? Here is the manuscript.

[gdetor]

Hi @brembs What do you mean by "good pipeline"? You mean how to convert GDoc to ReScience latex format?

[brembs]

Yes, that's what I meant, sorry. I'm writing nearly all my papers on GDocs these days as it is so very convenient, (especially in combination with PaperPile for referencing). It would be nice if there were a few simple steps to convert it from that format into whatever it is you need.

[gdetor]

@brembs You could use AbiWord and convert the GDoc to a .tex file. Then you can add the ReScience template packages at the preamble. Once you run a make (following the instructions on how to compile the article) the final pdf should be ready.

[brembs]

I'll try that,when it is time, thanks!

[rougier]

@brembs Any progress on the conversion?

[brembs]

I haven't done anything, yet, as we are still waiting for @benoit-girard to let us know if the manuscript should contain additional data, or if this one figure is sufficient. Once I know that the manuscript is (near) final, I'll try the suggestions here.

[gdetor]

@benoit-girard Gentle reminder

[benoit-girard]

Oh sorry! The notifications of this thread got lost in the middle of all ReScience notifications... I am trying to improve my workflow so that it does not happen anymore.

there are three additional figures (2 panels and one full figure) that I could immediately reproduce with the identical workflow as the one I used for figure 2 (I chose this figure because of the scientific content, which is important for my research and I still use this figure in teaching).

I feel these figures (those that can be reproduced without effort) should then be added to the manuscript.

There is one figure, Fig. 7, which I should be able to reproduce by writing another R-script that collects the relevant PIs from the 12 CSV files and plots them. Would be tedious, but the same principle as the other figures. For the remaining three figures (two panels and one full figure), one could try to run the old TP code on FreePascal. It may require some tweaking of the code, but my experience with FP (many years ago) was quite decent.

For the remaining figures, I would be nice to try to reproduce them, but I do not consider it as mandatory.

[gdetor]

@benoit-girard Thank you for the comments. @brembs Could you address reviewer's comments?

[brembs]

It is easy to just run the same procedure on more figures - what worked for one figure will work for the others. I'll generate new figures for them. I'll also have a look and see if I can get one other figure reproduced that isn't part of the same workflow, but I can't promise I will be able to get it done. All of this will take some time, but I don't know how much. I'm pretty much at capacity right now, so it will have to wait until I got some items off of my to-do list.

[gdetor]

Hi @brembs any progress on this matter?

[brembs]

It was looking good for a while late October, then manuscripts and theses needed revising and reviewing. Apparently, everybody also submits their manuscripts before the holidays, so I'm inundated with reviews and such. No chance I can get it done this year, sorry.

[rougier]

@gdetor Any progress?

[gdetor]

Hi @brembs Do you think we could finalize the process?

[brembs]

I've been trying, but still swamped. Hope to get it done before the summer holidays, which start in August here. No more teaching until then, so the chances look good.

[gdetor]

Hi @brembs Thank you for the update.

[gdetor]

@brembs Gentle reminder

[brembs]

I have been thinking that there was something last summer that I forgot to do :-) Yes, I've put it on my to-do list and will see that I get to it. Thanks for the reminder!

[rougier]

@gdetor @brembs Gentle reminder

[brembs]

Yes, yes :-)

[gdetor]

@brembs Happy new year. Gentle reminder

[brembs]

Thank you and happy new year to you, too! It is on my list and I am frustrated I wasn't able to get it done before the end of the break.

[gdetor]

@brembs Gentle reminder

[brembs]

Please believe me, I'm still trying. Difficulty lies in prioritizing this work, in part since the remaining work would only constitute a repetition of what I already showed for the first dataset.