Peer Review

[KennethEnevoldsen]

An overview of some of the problems are listed here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3237011/

Examples e.g. include 1) post-review, here articles are review post-hoc

Some of the big problems with peer-review is:

• lack of incentives for reviewers

[KennethEnevoldsen]

A potential approach would be that reviewers are added as co-authors.

Pros:

• primarily addresses: Give large incentives for doing review
• Ideally, a good reviewer makes meaningful scientific contributions to the paper which should be respected

Cons (and responses):

• Where in the author list should the review be placed (one approach is to do it unordered or mark roles, e.g. the reviewer would clearly be marked as such see #1 )
• Abusing reviews to get authorship: reviewer can "latch" onto good papers to gain credit. A potential solution for this is to simply allow the author to "accept" review. This would also mean that reviewers would have to be constructive to be considered a reviewer.
  • CON: This leads to it becoming hard to gain a bad review. How do we incentivize these?

@HLasse (saw that you starred the repo), @martinBernstorff (assumed you would like this, but feel free to say that you would prefer not to be tagged)

[KennethEnevoldsen]

Another solution that might co-occur with other solutions:

Use LLMs to auto-suggest changes to the text as a sort of pre-review (a sort of journal-based auto-correct built into the journal review process). This could also use some automated checks on citations and more.

Pros:

• Would make sure no stupid mistakes sneak into the papers
• Could help authors iterate upon their work without needing external recommendations
• Could act as a starting point for other reviewers
• Would be able to suggest papers for desk rejection based on official guidelines

Cons:

This would be an opt-in option for the review and for the desk reject part it would be up to an editor to make the final call. The biggest con is too many false positives.

[MartinBernstorff]

A potential approach would be that reviewers are added as co-authors.

Pros:

• primarily addresses: Give large incentives for doing review
• Ideally, a good reviewer makes meaningful scientific contributions to the paper which should be respected

Cons (and responses):

• Where in the author list should the review be placed (one approach is to do it unordered or mark roles, e.g. the reviewer would clearly be marked as such see Ordering Authors #1 )

• Abusing reviews to get authorship: reviewer can "latch" onto good papers to gain credit. A potential solution for this is to simply allow the author to "accept" review. This would also mean that reviewers would have to be constructive to be considered a reviewer.

  • CON: This leads to it becoming hard to gain a bad review. How do we incentivize these?

@HLasse (saw that you starred the repo), @MartinBernstorff (assumed you would like this, but feel free to say that you would prefer not to be tagged)

I think this would be excellent; incidentally, co-authors and reviewers should basically play the same role anyway (improving the manuscript and ensuring academic validity). Becoming a co-author of sorts also makes it very clear that you are, to some extent, responsible for the quality of what you have reviewed.

I like the "roles" model, and then perhaps placing reviewers in the middle of the review list to maintain "backwards-compatibility" with the traditional ordering.

[HLasse]

Yeah, +1 for the idea of adding reviewers as co-authors. I'm a big fan of the idea about assigning roles - it also seems to be something that at least some subparts of the ML field are pushing towards (fairly detailed author contribution lists).

[KennethEnevoldsen]

Good to hear and agree Lasse.

This also made me think of how to evaluate if a review is good? I see two main measures:

1) Author intrinsic evaluation: The author should find it relevant, informative etc. 2) Extrinsic evaluation: Peers should find that the review isn't "overly accommodating", "fair" etc.

1) is easy to measure in the system (author acceptance of the review), but 2) seems harder to both secure and measure