Reproducibility and reusability limitations in RegulatoryCircuits: analysis and solutions

[ghost]

Title: Reproducibility and reusability limitations in Regulatory Circuits: analysis and solutions

PDF URL: https://gitlab.com/mlouarn/RegulatoryCircuits/-/blob/main/Article/Louarn_ReScience_2021.pdf Metadata URL: https://gitlab.com/mlouarn/RegulatoryCircuits/-/blob/main/Article/metadata.yaml Code URL: https://gitlab.com/mlouarn/RegulatoryCircuits

Suggested editor:

Following our e-mail discussion, please find the submission of our work concerning a investigation of the Regulatory Circuit project methods and outputs. It mainly proposes an analysis of why the method could not be reproduced as well as an implementation to partially replicate the original results. It was not clear to us if this work was most suitable as a letter or a partially successful replication, so we provided the url for the code and we will change the article type if necessary.

co-author @mlouarn @odameron

[khinsen]

Thanks @fchatonn and @mlouarn for your submission! After a first reading of your paper, I think we can consider it either a (mostly failed) reproduction report, or a (mostly failed) replication report.

Before we go on, a question: it isn't completely clear to me, after a non-expert reading of your paper, what the status of your code is. I understand that it is a (partial?) implementation of your understanding of the original work - is that correct? If so, can you use it to replicate any of the intermediate or final results from the original work? In other words, is it possible to compare your work with the original work at the level of concrete results, in addition to a methodological comparison?

And also a comment: you refer to the Web site of the original work at http://regulatorycircuits.org, which as you say seems to have disappeared. There is, however, a relative recent snapshot (February 28, 2021) at the Internet Archive: https://web.archive.org/web/20210228000953/http://regulatorycircuits.org/. Would this help with your replication attempt?

[ghost]

We agree with you that our article could be considered as a mostly (or partially) failed replication report. Reproduction report seems less applicable since we tried to reproduce the methodology, but not with the code which was provided by the authors (java scripts).

To answer your question, yes our code is a partial implementation of our understanding of the original work, using both input data and intermediary results (after careful validation) used or provided by the authors. What our code and our work delivers is a replication of the original results, in some way corrected to what we think they should have been, if the described methodology had been strictly observed. We may also provide some additional intermediary results which were not available in the original work (i.e. individual scores per sample, which were aggregated to have the tissues scores, one tissue being constituted of several samples). We also tried to replicate and compare the intermediary results, but with no success due to discrepancies between the described and the actually used methodologies and to under-description of part of the methods.

Finally, thank you for providing the snapshot from the Internet Archive. This does not help us for our replication attempt, since we already downloaded all useful data from http://regulatorycircuits.org when it was still available; but this might be very useful for any other people interested in the Regulatory Circuit project. Indeed by following the links, we found that it is still possible to have access to the original results (http://www2.unil.ch/cbg/regulatorycircuits/FANTOM5_individual_networks.tar) and the supplementary data (www2.unil.ch/cbg/regulatorycircuits/Supplementary_data.zip) from the Lausanne University website, so we could indicate those links in our paper.

[khinsen]

Thanks @fchatonn for those explanations. Let's go for "replication" then!

@ctb Can you edit this submission?

[khinsen]

:bell: @ctb Can you edit this submission?

[khinsen]

@ReScience/editors Any volunteers for editing this submission?

[davan690]

Hi all, I can have a look but I am very novice when it comes to reviewing but am over my head in reproducibility so happy to have it a go. :) Ant

[rougier]

@khinsen @ReScience/editors Any volunteers ?

[khinsen]

@davan690 Do I understand correctly that you are volunteering to review this submission?

[khinsen]

@fchatonn Can you propose additional reviewers for your work? I'd then declare myself competent enough for the role of editor.

[ghost]

I sent a reply by e-mail, but I don't see it coming here, so I'm pasting it:

Dear all,

We can propose the following persons as suitable reviewers:

• Adrien Coulet, adrien.coulot@inria.fr
• Sarah Cohen Boulakia, cohen@lri.fr
• Ulf Leser, leser@informatik.hu-berlin.de

We hope that some of them will accept the reviewing work.

[odameron]

On 10/22/21 3:43 PM, fchatonn wrote:

I sent a reply by e-mail, but I don't see it coming here, so I'm pasting it:

Dear all,

We can propose the following persons as suitable reviewers:

• Adrien Coulet, @. @.>
• Sarah Cohen Boulakia, @. @.>
• Ulf Leser, @. @.>

We hope that some of them will accept the reviewing work.

Egon Willighagen (Univ. Maastricht) may be another relevant reviewer : egon​.​willighagen​@​​maastricht​university​.​nl https://www.maastrichtuniversity.nl/egon.willighagen

best wishes Olivier Dameron

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/57#issuecomment-949645004, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEM4QASAI236ZL5U6U7JGSDUIFTBNANCNFSM5BITG34A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[khinsen]

Thanks @fchatonn and @odameron for the reviewer suggestions! I'll start with the most active GitHub users.

[khinsen]

@egonw Would you be interested in reviewing a paper on the reproducibility of regulatory circuit analysis for the ReScience C journal?

[egonw]

@khinsen, this is too far outside my research field, I'm afraid.

[khinsen]

Thanks @egonw for your quick reply!

[khinsen]

@coulet Would you be interested in reviewing a paper on the reproducibility of regulatory circuit analysis for the ReScience C journal?

[davan690]

@davan690 Do I understand correctly that you are volunteering to review this submission?

Yes but I'm not sure this is the right way to do it @khinsen? I only just found this message in my notifications. Regards, Anthony

[khinsen]

@davan690 We do everything on GitHub as much as possible at ReScience, so yes, volunteering on an issue is very much the right way. You are herewith nominated reviewer! I am still looking for a second one, but you can start reviewing right now. If you have any questions about the review process, don't hesitate to ask!

[davan690]

@khinsen Thanks :) I am very new to the github issue workflow but I will do my best...and try and follow the instructions. Just to verify, I will start having a bit of a play around with the review this weekend :) cheers

[coulet]

@khinsen Hi, thanks for proposing the reviewing. I would be encline to accept, in particular with some guidance, as it would be my first with ReScience. I understood that it is mainly about verifying the replicability status. Is there reviewing guidelines? And a timeline for reviews? Best.

[khinsen]

@coulet Thanks for accepting to review this submission!

What we expect reviewers to evaluate is

1) Are the authors' claims on their results replicate (or not) the results of the original work justified? For example, if the authors say "our figure 1 is equivalent to figure 4 of the original work", do you agree? It's a matter of judgement that requires some familiarity with the subject domain.

2) Is the authors' code reproducible? Ideally, can you run it yourself and get the same results (figures, tables, ...)?

3) (as for other journals:) Is the paper clearly written?

Note that the goal of a ReScience C review is not to accept or reject a paper, but to improve it to the point of being acceptable. In this spirit, the review is a conversation between authors and reviewers, happening right here, for everyone to read.

If you have any further questions, just ask them right here as well!

[khinsen]

@davan690 @coulet Gentle reminder about your reviews!

[ghost]

@coulet @davan690 @khinsen Any news about the revision process? We're quite impatient to have a feedback on this work. Thank you, on the behalf of all authors.

[khinsen]

Sorry for the long wait @fchatonn! I lost track of this submission.

I contacted the two reviewers by e-mail to ask if they are still available for reviewing your submission.

[khinsen]

No reply from the reviewers... we will have to find two new reviewers!

[khinsen]

I contacted colleagues I know by e-mail, but nobody is available. So... I am turning to Twitter and Mastodon now.

[ghost]

@khinsen I am sorry, but after a discussion with the other authors, we decided it would be best to withdraw this contribution from ReScience and to try to have it published in some other journal. The first author is in her early researcher career and really needs both feedback about her work and publication records (we all do), so I hope that you will understand our decision.

Thank you again for taking your time to consider this contribution and please continue to contribute to better reproducibility and replicability of science!

On the behalf of all authors, Fabrice Chatonnet.

[khinsen]

@fchatonn No worry, yes, I understand your decision, and I even agree that our reviewing delay is way beyond acceptable - but our resources are limited.

I will close this issue, but note that it will continue to be visible. Your repository at GitLab will also remain visible, even if you delete it, as it has been archived by Software Heritage in the meantime. However, I think you can safely state that it's not under consideration by another journal when you submit elsewhere. Good luck!