2020/anonymity/

[utterances-bot]

Should the reviewers know who the authors are? - Hacking semantics

Why fully anonymous peer-review is important, and how we can achieve that in ACL rolling review reform.

https://hackingsemantics.xyz/2020/anonymity/

[cbrew]

Hiring, tenure and promotion committees should not use conference or journal acceptance rates as an index of quality. Nor should they use h-index. But they manifestly do in both cases. This is part of the problem, and it is out of ACL's control. Acceptance rates and and h-index are imperfect but entrenched methods for approximating the real goal, which is to provide a method for assessing the true quality and impact of research. The big advantage is that allow for a kind of approximate justice across different sub-fields of CS.

But that approximate justice comes at a cost: ACL and similar societies must provide venues with high-quality reviewing and below threshold acceptance rates. I submit that quality reviewing is something that the societies should want to do anyway, and that acceptance rate targets are something effectively imposed on the societies from outside.

So what if we give up on acceptance rates and find a better method of certifying the quality of our subfield's work? Suppose that this method is demonstrably better, but is not immediately adopted by other sub-fields. Are we disadvantaging NLP researchers relative to those in fields where acceptance rates continue to reign? Does this question have the same answer in the long term that it does in the short or medium term? Is this a Gresham's Law situation where bad research quality assurance practices crowd out potentially good ones?

I don't have solutions, but I think we must face the rather unpalatable facts about why we pay attention to acceptance rates. I wish we didn't have to, but think that we probably must.

[Isinlor]

I agree that using publication venue identity as an index of quality is wrong. There is actually a broader campaign to fight this entrenched practice the San Francisco Declaration on Research Assessment.

Of course, there are practical reasons to select papers for posters or presentations for some specific date when people gather together. But the idea that paper quality is binary and that this evaluation can be done once a year is just wrong. It doesn't help anyone besides committees.

The ideal situation would be just submitting papers for public discussion as soon as you are done with a paper. The paper would be public, but the paper authors would remain anonymous up until the first anonymous reviews are submitted, then the discussion could go on in a format that the discussion participants find the best. There is no need for an arbitrary binary accept reject system outside of practical limitations of in person gatherings.

On the other hand, my worst fear is that strictly enforcing anonymity creates periodic explosions of papers. There is no time to look at all these papers that suddenly appear, so from my perspective most of them get ignored even when published at some prestigious venue. They are lost in noise of an explosion and the biggest players overwhelm the small ones. Keeping publications naturally spread out trough the year allows to focus on more papers. I often feel like a dog chasing too many toys when such explosion occurs and in other periods it feels like there is not much happening at all because some black holes are sucking life out of community.

[allenschmaltz]

This is a great post. Incidentally, I wrote up my own response to the call for responses to the ACL proposal before seeing your post: https://github.com/allenschmaltz/Resolute_Resolutions/blob/master/volume1/volume1.pdf. In some sense, my view represents the opposite view with respect to anonymity, but we agree on a number of points (e.g., the dangers of free-for-all "PR review" by dropping peer review altogether), and in fact, some notion of "open" review could be used even with author anonymity. (Furthermore, between the two types of anonymity, author and reviewer identity, I think it's more important for science for reviewer identity to be non-anonymous, serving as an upward incentive for reviewers to put in the effort when completing reviews.) Everyone is driven by the same core motivations for reform: higher quality reviews, faster turn-arounds, guards against biases, etc., but the proposed means of getting there vary widely. For reasons I briefly discussed in my post, I remain skeptical that anonymity across the sets of interest (e.g., new researchers and incumbents) is reliably possible, given that even surface-level details leak important identity information in small subfields, so I think it's better for everything just to be completely non-anonymous, open, and transparent. This would also yield data for assessment: If there are bad actors/biases, it will [more likely than in the closed setting] become apparent via an analysis of the data, including at scale. (My guess is biases are actually far worse than generally perceived, since we want to know about the censored cases, but the relevant attributes/variables, such as the reviews, are largely hidden.) A post-publication review mechanism would also be helpful, in my opinion. The main downside of the open approach, perhaps, is that it's never been done like that before for NLP conferences, so there very well could be unforeseen (negative) consequences without trying it.