Proposed format for new paper summaries

[rando2]

Context As we begin the review, we need a place for contributors to track which papers have been read as well as a place to record summaries of the articles. This will help us to:

• Keep a list of older papers that may be relevant of new papers as they appear,
• Maintain an easy-to-access record of contributors' interpretation of the original lit
• Protect against authors' meaning from becoming distorted through the rapid editing of the main text by many contributors
• Provide a forum for discussion among contributors of the significance and interpretation of articles

Problem We need to make some sort of template to encourage the submission of papers as issues in a standardized format that will allow for filtering, searching, maintenance, etc. It would also be good to recommend contributors include certain pieces of information in their summaries.

Ideas

1. We likely want to use labels to differentiate new papers from other types of issues (e.g., people asking for help). Do we want separate labels for unclaimed vs claimed papers, or does the "assignees" function work to filter these out?
2. For searching, are user-provided keywords sufficient? Would we want to set up subject area labels, for example?
3. We could list questions to address in the summary as part of the template, but what if the person creating the new paper issue isn't the one who ends up reading/summarizing? How difficult would it be to set up a bot to comment on each New Paper issue, for example?
4. Below is my first pass at what information we might want users to provide. Feedback would be much appreciated!

Request thoughts from @cgreene

---

Please paste a link to the paper (preferably DOI) or citation information here:

Is this paper primarily relevant to Background, Diagnostics, or Therapeutics? (OK if more than one)

Please list some keywords (3-10) that help identify the relevance of this paper to COVID-19

Please leave a comment with your summary below. Suggested questions to address in summary:

• What did they analyze?
• What methods did they use?
• Does this paper study COVID-19, SARS-CoV-2, or a related disease and/or virus?
• What is the main finding (or a few main takeaways)?
• What does this paper tell us about the background and/or diagnostics/therapeutics for COVID-19 / SARS-CoV-2?
• Do you have any concerns about methodology or the interpretation of these results beyond this analysis?

[cgreene]

I am a big fan of issue templates. The OpenPBTA folks use these extensively. GitHub docs on the feature are good: https://help.github.com/en/github/building-a-strong-community/configuring-issue-templates-for-your-repository

For examples (probably too extensive to start with but as we learn how people are contributing we'll figure out how they should be designed) the AlexsLemonade/OpenPBTA-analysis repo has some.

[cgreene]

I only saw the top of this before. I like the bottom part. I think that at the beginning noting that it's ok to just post a paper and leave the other elements blank as prompts for folks who come to the issue later

[SiminaB]

Some of the points below are related to @juliettemarie0405's post in https://github.com/greenelab/covid19-review/issues/13. They're also based on my teaching a few lectures on evidence-based medicine and interpreting the scientific literature to medical and graduate students. :-)

One framework that is often used for assessing the literature is the "MAARIE" framework = Method, Assignment, Assessment, Results, Interpretation, Extrapolation (eg from http://cdn.journalism.cuny.edu/blogs.dir/422/files/2012/04/Different_types_studies_fromtextbook.pdf) I'm actually wondering if we should create a table for these elements? It may help to ensure some uniformity between reviewers. Depending on the number of papers, it may be helpful to have 2 reviewers per paper, even if we can't have a bona fide systematic review.

MAARIE framework: Method

• Exact question being investigated (can think of this through the prism of the null hypothesis being tested eg how many drugs/tests are being compared)
• Study population (eg sample size, country, age distribution, setting - patients presenting to clinic, random sample of school children etc)

Assignment

• Is the study observational or randomized? In either case, provide more details on the design, such as possible confounding variables, stratification, adequate masking (blinding) for randomized trials etc. I'm not sure if we're also including any animal/cell studies?

Assessment

• Is the measurement of the outcome appropriate?
• Are measurements complete? Eg is there a lot of loss-to-follow up? Are measurements subject to various kinds of bias? (eg if there isn't masking in randomized trials)

Results

• What methods are used for inference? Eg logistic regression, nonparametric methods etc
• Is there adjustment for possible confounders?
• What are the: estimated association, 95% confidence interval, p-value (or other measurements of statistical significance)

Interpretation

• Can we make a causal interpretation for the people in the study of test/drug -> outcome? Eg is it a clinical trial or does it match up with some of Bradford Hill criteria if it's not? https://www.edwardtufte.com/tufte/hill https://en.wikipedia.org/wiki/Bradford_Hill_criteria
• Are there identified adverse effects or interactions with other drugs?
• Are there specific subgroups with different findings? (Careful about multiple testing trap!)

Extrapolation

• Can we draw conclusions about other groups of individuals not specifically included in the study? For example:
• Do they extrapolate to similar groups?
• What would happen if the conditions are extended in terms of dose or duration?
• Can they be extrapolated to other populations or in very different settings? (eg different age group, primary care setting vs emergency department etc)

[rhagenson]

Adding my two cents as I see a possible solution to the problem of unclaimed/claimed papers. We could coerce the known pattern of stale versus active issues through the GitHub action stale into labeling issues that have not been labelled "claimed" as "unclaimed" after say 7 days. By default, this would be active on all issues unless we devise some exempt label, which I think opens us up to more complexity than just tracking which issues are actively being worked on by someone and which came in as suggestions then went unclaimed.

[cgreene]

I like this framing! @SiminaB, do you want to file a PR on the new paper template for this? It looks like it's going to be most appropriate for clinical trials. Is that correct? Should we start to distinguish issue templates by type of paper?

I think we're likely to have in vitro / animal studies as well since many of these compounds are at the very early stages of evaluation for this indication.

[SiminaB]

Will do! I don't think it will be for clinical trials only, but perhaps would make most sense for human studies.

[juliettemarie0405]

@cgreene @SiminaB

This isn't an exact partner to the terrific MAARIE tool, but an evaluative risk of bias tool specific for animal studies was developed by a Dutch group called SYRCLE (SYstematic Review Center for Laboratory animal Experimentation) based on the Cochrane tool- maybe helpful?

It asks the following questions:

Reporting Bias

1. Is it mentioned that the experiment was randomised?
2. Is it mentioned that the experiment was blinded (level unknown)?
3. Is a power / sample size calculation shown?

Selection Bias

1. Was the allocation sequence adequately generated and applied?
2. Were the groups similar at baseline or adjusted for confounders?
3. Was the allocation adequately concealed?

Performance Bias

1. Are the animals randomly housed during the experiment?
2. Were the caregivers/ investigators during the course of the experiment adequately blinded ?

Detection Bias

1. Were animals selected at random during outcome assessment?
2. Was the outcome assessment adequately blinded?

Attrition Bias

1. Were incomplete outcome data adequately addressed?

Other

1. Was the study apparently free of other problems that could cause a high risk of bias?

[rhagenson]

@cgreene I noticed you forked an issue renamer action that uses regexes to help organization, is there any plan to use a stale action to help ensure no paper/issue goes too long without being addressed?

[cgreene]

I don't have the bandwidth to do that right now but I could see it being helpful.

[rhagenson]

Thank you for your honesty. If I get the chance, I might draft something up in the coming days.

[cgreene]

There have been a whole bunch of efforts to improve this process. Where we are now is definitely than when this issue was filed. I'm going to close this so that it doesn't get scope creep. Please file new issues for new challenges as they arise! @rhagenson : thanks for considering it! 😁