fredgibbs
commented
8 years ago Thanks for bringing these up, Adam. I see both of these issues as distinct advantages of the open system rather than potential significant problems--provided, as you say, that people are nice to each other, which I presume will continue to be the case for our authors and editors. I know this is not always the case with other journals with more academic weight/prestige/gate-keeping behind them. But I'm proud that our community largely avoids these issues. I think the openness of our process actually encourages a more collegial and productive process than blind review (which can also be extremely useful; I don't mean to suggest it's always a bad experience).
1) I think reviewer and author negotiation is great! But at least as I see the editor's role, the reviewer isn't a gate keeper (that's explicitly stated on our review page), but more like a collaborator. As editors, I don't think we should ask anything to be reviewed that we don't think is pretty close to publishable. So lessons have already been accepted more or less and the review process makes them better through exposing more viewpoints and critical eyes to the lessons. This may need to change if we start getting more lessons than the editors can handle. I think the lessons that have been openly reviewed so far have gotten noticeably better, not worse, but we have a relatively small sample so far.
2) I can't think of a better way to encourage quality and timely reviews than by having the process be fully visible. But quick and thorough reviews won't always happen no matter what system is in place. But we also don't document the entire review process. We don't, for instance, note people who were asked to do a review but ultimately don't for whatever reason. So a review that takes a while get done isn't necessarily the fault of whomever ultimately submits the review. It might be, but there's no way of knowing how many people were asked beforehand or if the editor was slow, etc.
On Tue, Nov 3, 2015 at 2:53 AM, Adam Crymble notifications@github.com wrote:
I'd like to have an open conversation about the practice of open review we've adopted here, so I'd be grateful for ideas from a range of people. We've been doing open review for a few months now (reviewers openly post their comments with their names visible and authors respond).
There are a few changes between this and the traditional double blind review system used by most journals. In particular, I see the following possible matters that COULD become issues:
1) The role of the editor is diminished and the authors and reviewers might be tempted to negotiate between one another. This is fine if it is civil, but there is no opportunity for the editor to filter or otherwise mediate between two rapidly-responding individuals. There is also an inherent power relationship between the reviewer (a gate-keeper of sorts) and the author (trying to tick the boxes to get through the review process). If a reviewer exercises that power without the moderation of an editor, the lesson might actually get worse as the author seeks to tick the boxes and get 'done'. Is this a significant problem?
2) People who submit late reviews are clearly identifiable. Because reviews are dated when submitted to Github, it's really obvious who did a good thourough job in a reasonable amount of time, and who just didn't get around to it. That's frustrating for authors and editors, and it may also breed longer-term ill will between people. This was traditionally hidden by the editor who first collated responses. Is this a significant problem?
If you have opinions on these or other matters, please let us know, as it will inform our ongoing discussions.
— Reply to this email directly or view it on GitHub https://github.com/programminghistorian/jekyll/issues/148.
frederick w gibbs | assistant professor of history | univ. of new mexico fredgibbs.net | @fredgibbs http://www.twitter.com/fredgibbs
acrymble
drjwbaker
ianmilligan1
miriamposner
I'd like to have an open conversation about the practice of open review we've adopted here, so I'd be grateful for ideas from a range of people. We've been doing open review for a few months now (reviewers openly post their comments with their names visible and authors respond).
There are a few changes between this and the traditional double blind review system used by most journals. In particular, I see the following possible matters that COULD become issues:
1) The role of the editor is diminished and the authors and reviewers might be tempted to negotiate between one another. This is fine if it is civil, but there is no opportunity for the editor to filter or otherwise mediate between two rapidly-responding individuals. There is also an inherent power relationship between the reviewer (a gate-keeper of sorts) and the author (trying to tick the boxes to get through the review process). If a reviewer exercises that power without the moderation of an editor, the lesson might actually get worse as the author seeks to tick the boxes and get 'done'. Is this a significant problem?
2) People who submit late reviews are clearly identifiable. Because reviews are dated when submitted to Github, it's really obvious who did a good thourough job in a reasonable amount of time, and who just didn't get around to it. That's frustrating for authors and editors, and it may also breed longer-term ill will between people. This was traditionally hidden by the editor who first collated responses. Is this a significant problem?
If you have opinions on these or other matters, please let us know, as it will inform our ongoing discussions.