Revisions for Methods Manuscript

[rando2]

Review 1

• [x] 1.1 Issue: #1015
• [x] 1.2 PR: #1021
• [x] 1.3 PR: #1033, explanation in #1021
• [x] 1.4 Explanation in #1021
• [x] 1.5 Explanation in #1021
• [x] 1.6 PR: #1031, explanation in #1021
• [x] 1.7 Explanation in #1021

The authors have proposed and realized an interesting framework to allow for the construction of a review in collaborative setting while ensuring high levels of traceability. The idea is quite fresh, and the technical implementation appropriate for the back-end. Given the pioneering character of their work, I do not necessarily see its limitations as reasons for rejection, but as points where their experiences and challenges might by themselves be useful for others. Many concern the front-end.

1.1 Regretfully, their live review https://greenelab.github.io/covid19-review/ could deter possible readers, as – at least within the first ~5 minutes – it did not render robustly on two different computers of mine after scrolling (tried one M1 mac, and one i7/64GB mac).

1.2 Further the linked repository https://github.com/greenelab/covid19-review/tree/e5a4e3ce43f493c0c913ae647951488b89345106/output did not contain manuscript.pdf which would have allowed for a possibly smooth read (or more specifically: the link in the readme file would point to a file that would not be publicly visible) ; I thus strongly recommend prior acceptance that https://greenelab.github.io/covid19-review/ will be extended to allow users a download of the composite pdf without any need to scroll or recompile (and wait seconds/minutes for text to appear when advancing the screen at https://greenelab.github.io/covid19-review).

1.3 The main contributors to the automated review seem to have been people that also contribute as authors to the current submission, or are otherwise affiliated with them. For others planning community efforts, it could be useful if the authors shared an extended discussion on where they faced obstacles - and success - in the recruitment of contributors, and how they see their mobilization efforts to compare to other groups which try to build and promote community around COVID-19 literature (e.g.: subdivision of preLights). Additionally, none of the most frequent contributors appear to include a medical doctor from pulmonary medicine – suggesting that some interesting potential community members have not been attracted.

1.4 Presently the user-interface for the construction of the review is provided by git/github. As the authors note, this could deter some contributors. Would there be ways to create a simple user interface that appeals to target contributors?

1.5 Reading the automated review, it seems to aggregate information, rather than to synthesize and condense the literature – which is a role of reviews that has been noted by others before (e.g.: in “Laboratory Life” by Latour and Woolgar). Therefore, it would be interesting to learn more about the author’s intended target audience and usage scenarios – and whether the work that they created would be even something new aside from “reviews” or “literature surveys”.

1.6 Though thinkable given their technical solution for the back-end, the option for personalized reviews appears absent in the discussion.

1.7 For their data-integration on clinical trials, they could possibly use the integration provided by dimensions.ai to save redundant work on their end.

Review 2

See #1033 for all of these

• [x] 2.1
• [x] 2.2
• [x] 2.3
• [x] 2.4

This paper falls well into the scope of this workshop. It introduced a novel publishing platform, “Manubot,” which also acts as an infrastructure for collaboration and knowledge synthesis. In particular, it discussed the application of Manubot in producing a “living review” of covid-19, challenges that emerged with such a task, and how they dealt with the challenges. The paper is overall well written, and I only have a few minor suggestions.

(1) Section 2.1 didn’t give me a clear picture of the team structure that produces the review. What is the relation between recruited participants (“recruited by word of mouth and on Twitter”) and the consortium?

I am asking this also because, later on, you wrote: “We collaborated with the Immunology Institute at the Mount Sinai School of Medicine to incorporate summaries written by their students, post-docs, and faculty [49, 15]. Additionally, two of the consortium authors were undergraduate students recruited through the American Physician Scientist Association’s Virtual Summer Research Program.” It left me with the impression that there were two ways to recruit contributors. One is of relatively low barrier and informal (“recruited by word of mouth and on Twitter”). The other is of high barrier and formal (e.g., through organizations).

(2) Figure 3 has no associated narratives in the paper.

(3) It is not very clear to me how the knowledge synthesis and updating process take place. I only know how the summary statistics and figures were handled (“To address this concern, Manubot and GitHub’s CI features were used to create figures that integrated online data sources to respond to changes in the COVID-19 pandemic over time.”). How about the narratives?

(4) If you need more space to address the previous comments, I suggest the following to help you find some space.

• The spelling checking function (page 4-5) is not particularly interesting and doesn’t worth so much space.
• The emphasis that researchers in the biomedical domain are not particularly familiar with Git/GitHub. I understand it’s a major concern for the authors. Still, I don’t think this factor should take a front seat (e.g., mentioned at the very beginning of section 2.1) over other matters and be mentioned in several different places of the paper. I suggest pulling all materials about familiarizing users with GitHub into one section, including the concerns and solutions (e.g., training materials, easy tasks such as pull requests for beginners, using Gitter.im).

Review 3

See #1033 for all of these

• [x] 3.1
• [x] 3.2
• [x] 3.3
• [x] 3.4
• [x] 3.5
• [x] 3.6
• [x] 3.7
• [x] 3.8
• [x] 3.9

Summary This paper addresses the COVID-19 infodemic by proposing a collaborative manuscript writing approach, using Manubot. Researchers from a variety of domains were asked to contribute to a review article that organizes the large quantity of COVID-19 related research. To this end, Manubot has been extended to add functionalities specifically needed for this use case.

General remarks The paper is well written and relevant for this workshop, as it proposes a novel approach to collaborative sciences, in the form of version controlled living documents.

3.1 As the authors mention, the approach does require technical skills of researchers in order to contribute to a manuscript. Although instructions are provided, this is a major weakness of the approach. Although claimed differently by the authors, I do not think this approach will work for most researchers without the required technical skills. Even when they are willing to invest time in learning how to contribute to a manuscript, technical issues could arise and more complex git commands might be needed to solve this. What this paper is lacking, is a proper user evaluation that focuses on the question whether this approach is indeed suitable for non-technical users. For example, evaluating the usability of the whole process (e.g., with a SUS evaluation), and the participants’ attitudes towards the approach, ! or task load (via TLX assessment). At the current state, one cannot make claims on whether this approach is suitable for non-technical users.

3.2 In the introduction, the current challenges of scholarly communication (and in particular related to COVID-19 research) are listed. Although this paper does propose several solutions to mentioned issues (lacking collaboration, static review articles etc.), it does not directly address the most critical challenge: the "infodemic". The proposed approach still relies on document-based scholarly communication, and publications of the articles created with this approach will also contribute to the infodemic. A possible solution could be to integrate machine actionable descriptions of the results in the review articles, possibly by including semantic annotation to the text (I understand this might be out of scope for this research, but it is an interesting approach that would address the infodemic at its core).

Other remarks and questions

• 3.3 A comparison with other manuscript authoring approaches and tools would be interesting. Why was Manubot selected in the first place? Are there any alternatives available that do not require technical skills, but still have support for versioning and collaborative authoring? How would this approach compare to using a Wiki system for authoring? Using Manubot was taken for granted without a justification.
• 3.4 An itemized list of changes made to Manubot would be helpful (this is explained in the text already, but a list provides a better overview).
• 3.5 The selection of participants could be biased towards people that are interested in technology already, otherwise they would not have participated in the first place. This should be mentioned as limitation.
• 3.6 As mentioned in the paper, the manuscript grew too large for a single document. Why didn't the authors create multiple repositories for each of the articles?
• 3.7 When does a contributor become an author of a paper? For example, fixing a typo is presumably not enough to be considered a paper author?
• 3.8 Figure 5 does not seem to visualize the interests that are mention in the text and figure caption. Maybe the wrong figure is used here?
• 3.9 Consider using footnotes for URLs.

Other

• [x] Fix broken clinical trials reference in section https://greenelab.github.io/covid19-review/#recruitment-and-manuscript-development
• [x] Add GitHub workflow figure https://github.com/greenelab/covid19-review/pull/939#discussion_r619659243 #1026
• [x] Finalize and merge other figures in #944 and #953
• [x] Archive software (had problems with Software Heritage previously)
• [x] Discussion limitations of this approach from @rdvelazquez https://github.com/greenelab/covid19-review/pull/947#pullrequestreview-649162037
• [x] Correct @LucyMcGowan's name abbreviation (Lucy, should it be L. D’Agostino McGowan not L.D. McGowan?)
• [x] Post preprint to arXiv
• [x] Author agreement (CC BY 4.0, agitter signs on behalf of all authors)

[agitter]

Thanks for creating this @rando2. I have ideas for how to respond to many of these and can start creating a plan with you and the other co-authors.

I edited your post to add the TODO list I had for the camera ready version.

[agitter]

Software Heritage still can't archive this repository

Should we archive the master and external-resources branches on Zenodo? The goal would be to have a more permanent snapshot of our scripts that are described in the methods manuscript.

[rando2]

@agitter Zenodo sounds great to me. I think I have things set up so that we can generate the DOI when we create the release tomorrow.

[rando2]

Some remaining tasks for tonight/tomorrow, as far as I can see:

1. Add dynamic references to statistics and figures into the manuscript tomorrow, assuming #1034 is integrated as we're hoping
2. Confirm all figures have references in the text
3. Integrate LaTeX build with original submission (since @mprobson did a lot of finessing to get the layout right, it might be easier to keep that skeleton and just migrate our new text over).
4. Manually add the new dot plot from #944 and the new figure from #1026

[agitter]

Regarding the LaTeX build, @mprobson do you recall how much manual tweaking you had to do? I think it would be better to start from a fresh Overleaf document and use the old one as a reference for manual comparison. We changed the figures and a lot of text. It's easier to track the manual tex changes in a fresh Overleaf project as well.

I can create the project. I have a paid account, which exposes more of the history and tracks changes.

[rando2]

I actually might be thinking of the prior LaTeX document. He might not see this (he's on leave this semester so not always on email) but I'll check in with him when I get a chance!

[rando2]

Two updates:

1. Confirmed with @LucyMcGowan that it's L. D'Agostino McGowan
2. I think we might have fixed the broken clinical trials link? I'm not finding anything to fix anymore!

[agitter]

I submitted the methods manuscript to arXiv. The article is currently scheduled to be announced at Mon, 20 Sep 2021 00:00:00 GMT.

I think we might have fixed the broken clinical trials link? I'm not finding anything to fix anymore!

Yes, we must have fixed that

[agitter]

@rando2 I haven't tagged the latest arXiv/DISCO methods submission because I wanted to finalize what we want to do for the code in this repository. Should we link this repository to Zenodo? I think we would need to make a release of the master branch and another for the external resources branch to archive both. However, I've never tested archiving multiple branches in Zenodo before.

[rando2]

I think I have things set up correctly in Zenodo now, testing with this release for external-resources branch. It seems to work better in Safari than Firefox, for future reference (for myself 😆)

[agitter]

Thanks @rando2. It looks like the Zenodo archive worked perfectly.

Do you want to leave the Zenodo release archiving enabled so that even the other branch releases are archived? Or should we disable it now and re-enable it only if we want to make another external-resources branch release?

[rando2]

Do you want to leave the Zenodo release archiving enabled so that even the other branch releases are archived? Or should we disable it now and re-enable it only if we want to make another external-resources branch release?

I currently have it toggled for automatic preservation. I think it might be better to keep it on because this code determines the figures, which will vary by manuscript (for example, I need to look through the OWID dataset and get it integrated into the vaccines manuscript, which will probably cause some changes) -- but if it ends up causing a lot of overhead, we can definitely disable it!

[agitter]

It's okay with me to leave Zenodo archives enabled. I released pathogenesis-v3 as a test, and a new archive was created: https://zenodo.org/record/5534991

However, I've been the habit of tagging the output branch, which has the individual docx. That won't archive the figures on the external-resources branch directly, though it will save the versioned URLs that appear in the output documents.

We can continue discussing Zenodo archives here, but I'll close this now that the methods manuscript is done!