General guideline in citing non-peer-reviewed preprints?

[XieConnect]

What is our general guideline regarding citing preprints that have not gone through stringent peer review?

Should we avoid detailed citations/rephrased claims from preprints? e.g., should we only mention that some tasks have been tried by some preprints, but avoid detailed comparative/quantitative claims (e.g., "Preprint X has achieved much better accuracy/more effective than previous publications.") Some preprints seem to have flawed experimental design and evaluation. Thus their results may not be trustworthy. Thanks.

[agitter]

This should be discussed more broadly, but my opinion is that we should be careful and critical of all claims and evaluations in papers, peer reviewed or not. I don't recall which examples, but there was previous discussion of our skepticism about the method design and evaluation even in peer reviewed papers. In some cases, we have decided to not include a paper because we don't trust the results; in others, we might report the paper's claims but add own our caveats or editorializing.

To be more meta, in some ways our discussions here do constitute peer review.

Other opinions @cgreene?

[cgreene]

As I reviewed either peer reviewed or preprinted work, I tried to assess what was supported by the paper's results. If there were experimental flaws, I attempted to avoid discussing the work. I did, at one point, discuss potential overfitting in the experimental design in the imaging/categorize section. I think I may have also mentioned some existing literature that suggest that - at least in another field - any such overfitting is very minor.

I'd say it's our job to do thinking and that we can't just repeat what authors claim. We can either point out flaws or avoid discussion of claims that we think are flawed. If we see repeated examples of experimental design flaws (or potential flaws) we should probably have a subsection of the review that mentions them.

[cgreene]

Also specifically to:

Some preprints seem to have flawed experimental design and evaluation. Thus their results may not be trustworthy.

I think it might be helpful to note these issues in the github issue and send the authors a link to the issue asking them if they'd like to respond and offering to post an email response if desired. I have found that this has led to some really nice discussions.

If anybody would like to do this anonymously, feel free to send me the comment which I can post as from an anonymous participant. I'll also e-mail the authors and let them know that its there.

[cgreene]

Here is an example of just such a discussion:

https://github.com/greenelab/deep-review/issues/158

[agitter]

Conversing with the authors can definitely be a boon. That helped clarify some of my misconceptions in #110.

[cgreene]

Yea - #110 is a great example! @ueser was on github and able to quickly clarify.

[XieConnect]

Great suggestions and examples! I will pay attention to these rules during writing and reviewing. Thanks.