Regarding open review: overall, what went WRONG was....

[emhill]

I had a hard time finding the papers that were assigned to me in the submissions section, they didn't seem to be uploaded with consistent naming. Perhaps the papers could be uploaded to easy chair for review, but then any communication regarding the runnability of the artifact could happen as a github issue? That way the URL to the artifact submission could be put in the paper.

[mcmillco]

In reading the other reviews, I wonder if some reviewers didn't feel comfortable voting against acceptance while "exposed" publicly. I could see a situation in which a junior person feels uncomfortable pointing out flaws in a senior person's work.

[grammarware]

Yes, perhaps that could have been handled by collecting lists of pro et contra points and the chairs taking the blame themselves in calling each artefact as possibly accepted or rejected.

[emhill]

I'm confident enough that it doesn't bother me, but I would also think that gender or seniority could play a role in preventing people from speaking up if they can't get things working. (Of course it's the woman or more/less senior person that can't get our tool working....)

In this instance it seemed like we were really working to accept as many as possible, and we had a great group, so I don't think open reviewing is as big of an issue here. But if open reviewing is used for more competitive selection, it could also influence who's voice is most influential for accept/reject.

[JaneClelandHuang]

I think the problem that cmcmil mentions about people not feeling confident to vote publicly is a real issue. I also felt a bit sensitive to this issue in responding as an author. However - I think that the real issue here is that not so much is at stake for an artifacts track compared to a research track. If we want our Program Committees for leading research tracks to include younger researchers I think we have to be really sensitive to this issue.

Open reviewing seems to be diametrically opposed to double blind reviewing - which is the other current popular trend. So I wonder how well open reviewing would stack up when addressing the 'problems' of perceived bias that have been reported in recent ICSE PC reports. Wouldn't full-blown open reviewing just make this much worse?

Having said all that - I like open reviewing; however, I believe that it is going to lead to an increased perception of bias.

I think a better approach would be open anonymous reviewing. Authors and reviewers take on temporary personas. The exchange of ideas can be open - but nobody knows who the other people are. (..and sorry Tim - I put all this into a single comment - probably under the wrong discussion category).

[timm]

I think a better approach would be open anonymous reviewing.

it is possible to create >1 github identities.

if this idea takes on, can you imagine needing 1 anon per review process? once the reviewers send that anon id to the PC chairs (presumably via their real emails) then the pc chairs would know the "real id".

a halfway suggestion might be to suggest to folks involved in multiple open anon reivews to change their "anon" github idea every N months or so (say, N=4).

[JaneClelandHuang]

Tim - I don't know that github is really the place to do this though in the long-term. It isn't really designed for this - so using it is probably always going to include tweaking it. But anyway it might be amusing to discuss research with some quite interestingly named 'identities'. A reviewer could (for example) create quite varied moniker for themselves..

[timm]

@JaneClelandHuang : bags i "reviewer2"

[JaneClelandHuang]

so original :-)

[timm]

on a more serious note: i asked github for unlimited private repos in researchart. they agreed. so allowing one repo per paperSubmission and anon open reviewing then this might actual scale (caveat: need a little scripting to auto generate the repos from some data controlled by pc chairs)

[emhill]

I like this idea of the best of both worlds: triple blind with open-review-style author q&a. Now if only we can get technology to catch up to our ideals...

The reason why I think blinded reviewing is so important is that in theory it forces everyone to focus on the substance of the work, and not jump to conclusions based on who is speaking. The risk is that sometimes people are more critical & rude when anonymous in an open forum.

[timm]

and you would not be irritated in the complexities of joining such a pc? having to create an anon github id? (not really that complex i guess...)

[obaysal]

Using multiple channels for open review brought some confusion to reviewers. While we created a Google Doc page that showed assigned papers per reviewer (https://docs.google.com/spreadsheets/d/13sPyQvb7D1Scuof-1ERfqsKn0KNMUy-J2JJRtwsJkrQ/edit#gid=1872105011), as well as assigned issues (aka papers) to reviewers on GitHub, some reviewers could not easily find what papers are assigned to them. Not sure what is the best way here to make the review load more transparent.

[timm]

problem gets worse when if we use multiple private repos... reviewer load now spread out over n repos...

[obaysal]

I think all authors should provide their GitHub IDs and email addresses, so everyone is involved in the discussion (and gets notified of any comments) and not only the author that submitted the paper. Keeping all authors in the loop is important.

[timm]

if reviewers hide their id with anon github reviews, that is blind review

if reviewers and authors hide their id with anon github reviews, then that is double blind review

so, @obaysal u prefer blind review? or do i have ur position wrong?

meanwhile...

[grammarware]

We are thinking (with @kolovos) of using your model of artefact reviewing for SLE next month, and some ideas to tackle the anonymity of those who want it, include creating one anonymised account (since the terms of service of GitHub do not allow shared accounts, it will have to be maintained by PC chairs and the comments would have to go through them), many anonymised accounts (one per actual person, as proposed above) or separating comments (posted openly) and judgement scores (sent privately).

I am an old hacker and therefore think that all information must be free, but I do understand and acknowledge that openness in sensitive places like reviewing may lead to conflicts and may affect grumpy senior researchers' opinions of their younger counterparts ;) As chairs of such committees, of course it is our duty to make sure participating in a review process is not harmful for them.

[timm]

we need to be v.mindful of the issues raised by our female colleagues who report significant concerns of gender bias in reviewing. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4552397/ so going to anon open review seems a useful step

as to being an old hacker... me too. but "information must be free" in think is an adage we might want to revisit. if ze old days of net programming equality there was no harm in all telling all. no power structures (yet) in a 1980s style unix site. since then, EVERYTHING is on line-- including reputations that can be damaged and not repaired. so i would say information should be free, within the boundaries on good manners and kindness and the right to privacy if u want it.

[timm]

and here's a happy doggy, just cause i want too...