Review Pisani & Botchway's "Sharing individual patient and parasite-level data through the WorldWide Antimalarial Resistance Network platform: A qualitative case study"

[Daniel-Mietchen]

Earlier this month, I was contacted by Wellcome Open Research (which uses public post-publication peer review) about reviewing the following paper:

Pisani E and Botchway S. Sharing individual patient and parasite-level data through the WorldWide Antimalarial Resistance Network platform: A qualitative case study [version 1]. Wellcome Open Res 2017, 2:63 (doi: 10.12688/wellcomeopenres.12259.1) .

By then, I had already had a first read of the paper but wanted to dig deeper into it anyway, so I was inclined to agree to the assignment. However, they wanted the review within a week, to which I said no because I was traveling, and offered to do it by the end of the month instead, which was OK with them. I got a reminder from them this week, and the end of the month is now — thankfully with a slice from the weekend, which makes it easier to find some quiet time for this kind of activity.

[Daniel-Mietchen]

I actually started the review process by re-reading the paper on Wednesday while in the waiting room at my doctor's. This means I read it on my phone — an experience that could certainly do with some improvements, for which I have just opened #495.

I then followed up late on Friday by going through all the associated data files (which I hadn't looked at when I first read the paper), in which I found some bits that addressed some of the things I wanted to dig into, e.g. the set of 78 papers from which patient records were included in the WorldWide Antimalarial Resistance Network (WWARN) database, according to the Pisani & Botchway paper (I'll refer to it as P&B in the following). This has triggered #491.

[Daniel-Mietchen]

Just noticed that the

e.g. the set of 78 papers from which patient records were included in the WorldWide Antimalarial Resistance Network (WWARN) database

part above is misleading.

Comments on GitHub don't have the git versioning that is so useful for the main content of the site (i.e. the repos), so it's hard to track changes in comments, and I am thus using this extra comment to explain that what I meant above is something more like

e.g. the set of 78 papers that included patient records from the 
WorldWide Antimalarial Resistance Network (WWARN) database

.

[Daniel-Mietchen]

Also earlier this week, I had a look at the review form that Wellcome Open Research (WOR from now on) are using. This meant I had to register on their site, which was possible through ORCID in principle, but not yet in practice, since

• the email they had used to contact me wasn't yet registered with ORCID, so I did that
• the email display options had been set to "visible only to me" (probably the default — I don't remember having fiddled with that setting before), so I had to change that to "visible to trusted parties", which includes sites like WOR for which I had given permission to log in via my ORCID.

Then, I could see the reviewer form, which is structured as follows: After some basic article metadata, there is a section "Your Report", which states "Your referee report and your name and affiliation (and those of any co-referees) will be published alongside the article, and publicly visible to all. For more information, see Referee Guidelines."

I haven't had a look at the Referee Guidelines yet but will do so before I actually start the review.

First, here are some more bits from the reviewer form:

• Is the work clearly and accurately presented and does it cite the current literature? [ ] Yes [ ] No [ ] Partly

• Is the study design appropriate and is the work technically sound? [ ] Yes [ ] No [ ] Partly

• Are sufficient details of methods and analysis provided to allow replication by others? [ ] Yes [ ] No [ ] Partly

• If applicable, is the statistical analysis and its interpretation appropriate? [ ] Yes [ ] No [ ] Partly [ ] Not applicable [ ] I cannot comment. A qualified statistician is required.

• Are all the source data underlying the results available to ensure full reproducibility? [ ] Yes [ ] No [ ] Partly [ ] No source data required

• Are the conclusions drawn adequately supported by the results? [ ] Yes [ ] No [ ] Partly

• Please provide a full report, expanding on your answers to the questions above. In particular, if you answered “no” or “partly” to any of the questions, please give constructive and specific details as to how the authors can address any criticisms. Please indicate clearly which points must be addressed to make the article scientifically sound.

Below that is a free-text field that says "Write your report here (minimum of 50 words)..." — the text actually went away when I went in there to copy it from there to here, but reloading the page confirmed the text my memory had served me.

Below that is a form to add a citation (briefly tested that with the DOI of this paper, and it seemed to work nicely), and then comes a "NEXT" button and a "SAVE & RETURN LATER" button, all of which inactive, presumably because I have not yet responded to the questions and met the 50 words minimum.

Below those buttons, other sections can be seen only via their headings, so I don't know yet what will be in there in detail. They are:

• Article Approval status

• Your Details

• Competing Interests

[Daniel-Mietchen]

I have now had a look at the Referee Guidelines.

They

• have a rather busy landing page ( https://wellcomeopenresearch.org/for-referees/guidelines , same as above) not very conducive to actually being read
• link forward to
  • a How it Works page that I had seen when they launched — not much different from F1000 or RIO, so won't give it a detailed read, except for the "Referee Criteria" and "The Referee's Role" sections
  • "The Referee's Role" links to "guidelines specific to each article type" (yes, a PDF), and the article type here is "Research Articles", for which the guidelines state

    • "Research articles should present original findings, such as results of basic and translational research, clinical and epidemiologic studies, or clinical trials, or the outcomes of research projects in social sciences and humanities. Null and negative findings and reanalyses of previous studies leading to new results, as well as confirmatory results, are all suitable. The peer review should focus on whether the paper is fully scientifically sound, not on the likely impact of the work."

    • the above-mentioned questions (with a typo)
  • policies
  • the generic Policies page https://wellcomeopenresearch.org/about/policies
    • subsections thereof (e.g. https://wellcomeopenresearch.org/about/policies#ethpol for ethics, even though the link from https://wellcomeopenresearch.org/for-referees/guidelines misses the #ethpol part)
  • guidelines
    • for finding referees https://wellcomeopenresearch.org/for-authors/tips-for-finding-referees
    • for sharing data https://wellcomeopenresearch.org/for-authors/data-guidelines

I'll look these things up again as they come up in the review form, and comment on them in more detail then.

From here on, the further plans are:

• read the article again (on my computer) and annotate it via http://hypothes.is/
• also annotate the dataset to the extent feasible
• continue to document the process here
• write the reviewer report around those annotations, and add a summary as well as a link to this review ticket for background
• hopefully close the ticket while it's still September anywhere on Earth

[Daniel-Mietchen]

I'm going to start annotating the paper — progress can be followed via https://via.hypothes.is/https://wellcomeopenresearch.org/articles/2-63/v1 .

[Daniel-Mietchen]

I'm done annotating but too tired to write the report now.

[Daniel-Mietchen]

I'm back on, starting to write the report now.

[Daniel-Mietchen]

Here is my first draft of my review:

This article investigates the sociocultural context of the establishment, design and development of a global epidemiological data sharing initiative — the WorldWide Antimalarial Resistance Network (WWARN) — from its beginnings over a decade ago until about a year ago.

It highlights that

• data sharing requires dedicated infrastructure, including relevant standards and standard-aware tools
• designing this infrastructure and the associated policies requires engagement with all relevant stakeholders
• developing the above takes time and requires stable funding, including for the stakeholders involved
• buy-in from stakeholders additionally requires significant cultural changes in terms of how data sharing and the reuse of shared data and tools are valued.

None of these points are particularly new, but the strength of the paper lies in weaving in the often-neglected concept of equity, e.g. by discussing the need for capacity building in malaria-endemic countries (i.e. mostly in lower and middle income settings) in terms of analyzing data pooled across multiple countries, the various direct and indirect interactions between single-country and multi-country studies, the role of mediators like the World Health Organization, the role of WWARN study sections as well as some generic and specific inequalities in research funding, publishing and assessment.

The three main sources of the information presented are

• a set of documents from WWARN and its partners as well as individuals involved
• a set of interviews with individuals who are or have been involved with WWARN internally or externally
• a witness seminar that brought some of those individuals together to discuss themes emerging from preliminary analysis of the documents and the set of individual interviews.

Basically none of these key sources have been shared (for justified privacy reasons), which naturally limits the reproducibility of the study. However, the authors document their sources' metadata in sufficient detail to allow the reader to follow and evaluate the logic of the overall narrative as well as some of its highlighted details, e.g. the differences of perspectives of the different stakeholders, and the change of focus of the network from policy-oriented to research-oriented.

The study was qualitative in nature, so there is basically no statistical analysis. It also looked at the WWARN case study almost in isolation, which limits the generalizability of the conclusions in principle, but as WWARN's approach itself is now being generalized to other epidemiological use cases in the framework of the newly established Infectious Diseases Data Observatory (IDDO, a WWARN umbrella), this provides fertile ground for testing the validity of the paper's conclusions in other epidemiological contexts like schistosomiasis, Ebola and visceral leishmaniasis.

Overall, I found the paper well written, and in the few cases where I found something that would benefit from corrections, additions or clarification, I left a comment as part of my annotations of the paper, which are accessible via https://via.hypothes.is/https://wellcomeopenresearch.org/articles/2-63/v1 . None of these would preclude approval here in Wellcome Open Research, but I would still like to encourage the authors to address them in a revised version of the manuscript.

One point I did not mention in my annotations is the lack of visuals. While not essential, it would have been useful to have some visual representation of the kinds of data that are now shared through WWARN/ IDDO, as well as of the way(s) in which they were shared (privately or more broadly) before WWARN, over the course of WWARN development and now through the WWARN/ IDDO system. This can help the reader, but it can also help to attract new readers from within or near the paper's target group of "researchers who share data, or are contemplating doing so".

For my notes on drafting this review, see https://github.com/Daniel-Mietchen/ideas/issues/494 .

I will now go and read the first review that has already been published, and then reconsider mine in that light.

[Daniel-Mietchen]

That was a quick read - that review is just a basic nod to the paper. Despite being short, it manages to resonate with my own assessment of the paper, except for the

Are all the source data underlying the results available to ensure full reproducibility?

part, to which he responded yes, while I will respond "partly", for reasons outlined above.

I will now give my review a final read and then go back to the form to fill in whatever is still required (I don't like it that I can not see the entire form(s), which makes it harder to plan the time it will take to fill things in).

[Daniel-Mietchen]

In the following comment, I will be pasting the final version of my review, right before pasting it into the form over at WOR.

[Daniel-Mietchen]

This article investigates the sociocultural context of the establishment, design and development of a global epidemiological data sharing initiative — the WorldWide Antimalarial Resistance Network (WWARN) — from its beginnings over a decade ago until about a year ago.

It highlights that

• data sharing requires dedicated infrastructure, including relevant standards and standard-aware tools
• designing this infrastructure and the associated policies requires engagement with all relevant stakeholders
• developing the above takes time and requires stable funding, including for the stakeholders involved
• buy-in from stakeholders additionally requires significant cultural changes within the research and policy ecosystem in terms of how data sharing and the reuse of shared data and tools are valued (especially in epidemiology, but also beyond).

None of these points are particularly new, but spelling them out on the basis of a concrete example like WWARN and enriching them with sound bites — e.g. “Seven years ago, "data sharing" was a swear word.” — is useful both for conversations within a given field (in this case epidemiology) and across fields.

The main strength of the paper lies in weaving in the often-neglected concept of equity, e.g. by giving voice to representatives from malaria-endemic countries (i.e. mostly in lower and middle income settings) and by discussing

• the need for capacity building in such countries in terms of analyzing and otherwise utilizing data pooled across multiple countries
• the various direct and indirect interactions between single-country and multi-country studies
• the role of mediators like the World Health Organization
• the role of WWARN study sections as well as
• some generic and specific inequalities in research funding, publishing and assessment.

The three main sources of the information presented are

• a set of documents from WWARN and its partners as well as individuals involved
• a set of interviews with individuals who are or have been involved with WWARN internally or externally
• a witness seminar that brought some of those individuals together to discuss themes emerging from preliminary analysis of the documents and the set of individual interviews.

Basically none of these key sources have been shared (for justified privacy reasons), which naturally limits the reproducibility of the study. However, the authors document their sources' metadata in sufficient detail to allow the reader to follow and evaluate the logic of the overall narrative as well as some of its highlighted details, e.g. the differences of perspectives of the different stakeholders, and the change of focus of the network from policy-oriented to research-oriented.

The study was qualitative in nature, so there is basically no statistical analysis. It also looked at the WWARN case study almost in isolation, which limits the generalizability of the conclusions in principle, but as WWARN's approach itself is now being generalized to other epidemiological use cases in the framework of the newly established Infectious Diseases Data Observatory (IDDO, a WWARN umbrella), this provides fertile ground for testing the validity of the paper's conclusions in other epidemiological contexts like schistosomiasis, Ebola and visceral leishmaniasis.

Overall, I found the paper well written, and in the few cases where I found something that would benefit from corrections, additions or clarification, I left a comment as part of my annotations of the paper, which are accessible via https://via.hypothes.is/https://wellcomeopenresearch.org/articles/2-63/v1 . None of these would preclude approval here in Wellcome Open Research, but I would still like to encourage the authors to address them in a revised version of the manuscript.

One point I did not mention in my annotations is the lack of visuals. While not essential, it would have been useful to have some visual representation of the kinds of data that are now shared through WWARN/ IDDO, as well as of the way(s) in which they were shared (privately or more broadly) before WWARN, over the course of WWARN development and now through the WWARN/ IDDO system. This can help the reader, but it can also help to attract new readers from within or near the paper's target group of "researchers who share data, or are contemplating doing so".

For my notes on drafting this review, see https://github.com/Daniel-Mietchen/ideas/issues/494 .

This review is licensed CC0, as per http://creativecommons.org/publicdomain/zero/1.0/deed.en .

[Daniel-Mietchen]

After hitting the "NEXT" button, a new screen appears, with the following form fields:

Article Approval status

Please select one status: Approved

[ ] No or only minor changes are required. The experimental design, including controls and methods, is adequate; results are presented accurately and the conclusions are justified and supported by the data.

[ ] Approved with Reservations The article is not fully scientifically sound in its current version, but the criticisms could be addressed with specific, sometimes major, revisions.

[ ] Not Approved The article is of very poor quality and there are fundamental flaws in the article that seriously undermine the findings and conclusions.

I went for "Approved".

Your Area(s) of Research (optional) Please state your area(s) of expertise (e.g. immunogenetics, endocrine disorders, single-cell technologies), particularly if you feel that you are able to assess only certain aspects of this article.

Here, I went for "Data sharing policies and infrastructure".

[Daniel-Mietchen]

After hitting the "NEXT" button again, they asked for my ORCID and affiliation, which I simply filled in. Since I had used my ORCID for signing in, it already appeared there, and even the affiliation was almost correct (though not sure where they got it from, since it's not yet in ORCID — just opened #500 on updating that), so this screen took just seconds to get past.

[Daniel-Mietchen]

Competing Interests Do you, or any named co-referee, have any competing interests that you need to declare (given that your report will be open to readers) with respect to this article and its contents? [ ] No (the line "No competing interests were disclosed" will be added to your report) [ ] Yes

The term "competing interests" was linked to https://wellcomeopenresearch.org/resources/WellcomeOpenResearch-Referee-Competing-Interests.pdf (yes, a PDF again, for no obvious reason), which had the following

Declaration of Competing Interests for Referees

We ask that all referees disclose both 'Non‐Financial' and 'Financial' Competing Interests that might lead a reasonable person to question whether your interpretation of the data or of the article may have been influenced by your personal or financial relationship with other people or organizations. For every referee report submission, you must state whether you have any competing interests, and if you disclose that you do have some, you must provide details.

All competing interests that are declared will be displayed against your referee report. If no competing interests are provided, the line: ‘No competing interests were disclosed’ will be added to your report. If you are unsure whether you have a competing interest, please contact our editorial office at editorial@wellcomeopenresearch.org.

When deciding if you have a competing interest, it might be helpful to consider the following examples, but note that this is not an exhaustive list:

Financial competing interests

• In the past five years have you received reimbursements, fees, funding, or salary from an organization that may in any way gain or lose financially from the online listing of this work, either now or in the future? Is such an organization financing the work presented in this article or its presentation at a conference? If so, please specify.
• Do you hold any stocks or shares in an organization that may in any way gain or lose financially from the online listing of this work, either now or in the future? If so, please specify.
• Do you hold or are you currently applying for any patents relating to the content of the work? Have you received reimbursements, fees, funding, or salary from an organization that holds or has applied for patents relating to the content of the work? If so, please specify.
• Do you have any other financial competing interests? If so, please specify.

Non‐financial competing interests

• Are you a current collaborator with any of the authors of the paper in question, or have you been in the past 3 years?
• Have you co‐authored a paper with any of the authors of the paper in question in the past 3 years?
• Are there any other non‐financial competing interests (political, personal, religious, ideological, academic, intellectual, commercial or any other) to declare in relation to your assessment of this work? If so, please specify.

I clicked "Yes", which created a new form field

Enter details of the competing interests

Here, I put in the following:

I have been in contact with the study's first author since May this year, as there is some overlap of our respective work with regards to data sharing in public health emergencies. She is also involved in the organization of a Wellcome-funded workshop on data sharing in low and middle income countries later this month, which I am planning to attend.

The form continues as follows:

Declarations Wellcome Open Research is a service to Wellcome Trust researchers and grant recipients provided by F1000 Research Limited (“the Service Provider”) on behalf of the Wellcome Trust (“Wellcome”). By submitting your referee report (“the Report”) you warrant and represent on behalf of yourself and your co-referees that:

• You are authorized by your co-referees to enter into this agreement.
• You are the sole author(s) of the Report and the sole owner(s) of the copyright in the Report.
• Nothing in the Work is obscene, defamatory, or libellous, violates any right of privacy or infringes any intellectual property rights or any other human, personal or other rights of any person or entity or is otherwise unlawful.
• You grant to Wellcome and the Service Provider the non-exclusive right to publish, reproduce, distribute, display and store the Report together with our names and affiliations.
• I and my co-referees (if any) authorize the use of the Report in accordance with the Creative Commons CC BY license.

I have put my text under CC0, which is compatible with them putting a "Creative Commons CC BY license" (the terms is linked to https://creativecommons.org/licenses/by/4.0/ , i.e. version 4.0) on it.

[Daniel-Mietchen]

I then hit the "PREVIEW" button, which brought me to a preview screen that had the expected information plus a statement

Referee Declaration I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

(which I think should be more prominent) and a note (in Italics)

Once you have submitted your report, we will publish all information you see on this Preview page, including your name and affiliation. In line with our open peer review policy, this information will be visible to all readers. The editorial team may make minor edits to referee reports to correct typos or improve clarity.

Next to that note, there was an option to go back to edit my responses. I went back just to try it out but did not change anything. Upon hitting the "SUBMIT" button, a confirmation message popped up, asking whether I'm sure I want to submit this report, which I confirmed.

[Daniel-Mietchen]

The next screen said

Thank you for submitting your report Our editorial team will be in touch shortly to let you know how you can view your published report and get credit for your work as a reviewer.

I reloaded the paper and did indeed not see my review there yet, so I suppose they will do some editorial review over the next few days.

While I did that, an email came in thanking me as well.

I'll leave the ticket open until the review appears.

As an aside, I'll note the current metrics:

• 147 views for the paper (dozens of these were mine, probably even if they merged traffic by IP or cookies, since I have accessed the paper from multiple different places and browsers)
• 36 (PDF) downloads (none of them mine, though now that I am writing this, I am wondering whether I should have checked the PDF as well...)
• 6 views for the first review (the last one of them was mine)

[Daniel-Mietchen]

Not counting the two initial readings of the paper and the inspection of the data supplement, doing this review — including the annotation of the paper and the documentation of the process here — took me about 9h overall, distributed over three weekend/ holiday nights.

I have never documented the process in such detail, and most of my reviews so far have taken less time, but some with large or complex datasets have taken longer.

[Daniel-Mietchen]

WOR got back to me inquiring about the nature of the interaction I had had with the paper's first author, to which I replied with further details that I cannot make public because it's about another project of hers, on which there is no information publicly available yet. I surely would have preferred to simply link to the project.

They concluded that

we agree with you that this does not represent a significant collaboration and we will go ahead with publishing your report

The review has now been posted on the WOR site, and they recommend the following citation:

Mietchen D. Referee Report For: Sharing individual patient and parasite-level data through the WorldWide Antimalarial Resistance Network platform: A qualitative case study [version 1; referees: 2 approved]. Wellcome Open Res 2017, 2:63 (doi: 10.21956/wellcomeopenres.13272.r25804) The direct URL for this report is: https://wellcomeopenresearch.org/articles/2-63/v1#referee-response-25804

I am thus closing this ticket.