discussion format of author response

nschneid commented 1 month ago

The discussion format of the author response period is mentioned as a new feature of the Oct 2023 cycle, but there is no elaboration on whether it is a long-term design choice or how, substantively, people are supposed to engage in it (what kinds of interactions are suggested?).

https://x.com/complingy/status/1820239363222966677

jkkummerfeld commented 1 month ago

I believe this was a request made by 2024 PCs.

Adding guidance on interaction is a good idea. I'll pass this on.

jmhessel commented 1 month ago

Thanks for kicking off this discussion @nschneid and @jkkummerfeld ! I didn't know about this thread, and I tweeted about this concern recently. Following my tweet, I sent this mail to editors@ , which I will repost here for visibility:

Hello!

Thank you for your work on rolling review! My name is Jack Hessel, and I have been doing service for *ACL/EMNLP for a while, including reviewing, ACing, SACing, etc.

One hurdle I have encountered in this cycle for ARR (where I am SACing for EMNLP) is: folks are not clear on what is versus is not allowed in the author response. A common situation:

1. Reviewers point out missing experiments.
2. Authors responded with significant new experiments that address the requests.

Are the new results allowed to be considered for the current cycle, or not? I also made a twitter thread about this to get folks' perspective: it seems, in general, this is not 100% clear. https://x.com/jmhessel/status/1823436524639412591

The ARR website says: "The intention of this is to clarify any misunderstandings or confusions on the part of the reviewers (not to reargue the case for your paper)." but it does not specify a rule ("the intention is...") nor is it precise (are new experiments allowed, or are new experiments merely clarifications?). This leads to situations where some ACs allow significant new results, and some do not. I am doing some emergency AC work, and I am not sure what policy to apply.

This ambiguity also seems to be affecting conference PCs as well. For example, the EMNLP PC sent a slightly more specific set of clarifications:

"The rebuttal gives you an opportunity to clarify any misunderstandings or confusions on the part of the reviewers. It is not an opportunity for you to reargue the case for your paper *or improve your paper.*" (emphasis on what seems to be a slightly more specific, but ad-hoc for EMNLP 2024, clarification)

But, this guidance 1) is only available in an email called "[ARR June / EMNLP 2024] Author response: some clarifications"; and 2) still does not precisely specify whether significant new experimental results can be included or not.

Would it be possible to add more precise clarification for AC/SAC about what author response components can be considered in assessment for a given cycle versus not? I am happy to provide some draft wording to be considered if that would be helpful.

(If it's helpful to look to peer venues as a starting point, PAMI-TC's policy adopted by CVPR, is slightly more specific: 

>  Authors should refrain from including new contributions or experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers. Reviewers are instructed to disregard any such contributions.

https://cvpr.thecvf.com/Conferences/2024/AuthorGuidelines)

Jack

annargrs commented 4 weeks ago

The author response instructions are now updated on the website: https://aclrollingreview.org/authors#step2

TLDR: minor new experiments are allowed (e.g. a new ablation, or hyperparameter setting, or an extra baseline), but only minor, and only directly in response to reviewer question.

nschneid commented 4 weeks ago

Helpful guidance, thanks!

I have also encountered several author questions about etiquette in interactions during the author response period + commitment steps:

If a reviewer doesn't respond promptly to an author response addressing their concerns, is it a good idea to post a followup reminder? Or flag it in a confidential comment to the AC?
Is it a good idea to press reviewers to raise their scores if I believe my response addressed their concerns? If they don't, is that grounds for a confidential comment to the AC?
Is it bad form to leave the "optional" commitment comment blank? Should it be used for an assurance that reviewer/metareviewer suggestions will be heeded in the camera-ready?
If a metareview mentions weaknesses/suggestions raised by reviewers but does not explicitly acknowledge the author responses regarding those suggestions, should this be flagged to SACs, or in a commitment comment?

I don't know if every aspect of etiquette can/should be addressed with official guidance—maybe this is the sort of thing that develops in the community over time. But it is notable that (esp. junior) authors worry about these things.

jmhessel commented 4 weeks ago

Thanks @annargrs ! Regarding new experiments --- this new guidance is much more specific, and would definitely be helpful for me as an AC.

acl-org / aclrollingreview

discussion format of author response #132