Number of deadlines per year

[parno]

In Issue #7, @danwallach says:

We initially scheduled JETS with four submission deadlines. This made things remarkably tight so we shifted to three, which felt much more reasonable. Here's the schedule that the USENIX production staff and we ultimately converged on:

Deadline 1: Submissions due: Tuesday, August 6, 2013, 11:59 p.m. PDT Initial notification to authors: Tuesday, September 10, 2013 Second-round revisions due: Tuesday, October 1, 2013, 11:59 p.m. PDT Final notification to authors: Tuesday, October 29, 2013 Final files due: Tuesday, November 12, 2013

Deadline 2: Submissions due: Thursday, December 5, 2013, 11:59 p.m. PST Initial notification to authors: Thursday, January 9, 2014 Second-round revisions due: Thursday, January 30, 2014, 11:59 p.m. PST Final notification to authors: Thursday, February 27, 2014 Final files due: Thursday, March 13, 2014

Deadline 3: Submissions due: Tuesday, April 8, 2014, 11:59 p.m. PDT Initial notification to authors: Tuesday, May 13, 2014 Second-round revisions due: Tuesday, June 3, 2014, 11:59 p.m. PDT Final notification to authors: Tuesday, July 1, 2014 Final files due: Tuesday, July 15, 2014

The gist of this was that we had three deadlines, wherein we assigned reviewers and went through the usual reviewing process. A month later, we'd have the first round of decisions, which had one important gate: either you're in (done!), you need to make minor revisions (due in a month), or you need to make major revisions (and you would then resubmit to the next deadline). In practice, the question of "do you think the authors can address your concerns in one month?" served to help resolve which path was appropriate for each paper.

Also, this process played relatively nicely with HotCRP, wherein we had a new HotCRP instance three times per year. Eddie Kohler added a feature for me that made it easy to dump the reviewer database (including area preferences) and load it into subsequent HotCRP instances. HotCRP otherwise isn't set up to manage the sort of dataflow typical of an open-ended journal (which may or may not be an issue for S&P).

Keeping reviewers on task and hitting deadlines. I had to do a fair bit of browbeating to ensure that we hit all these deadlines, which is part of why we simplified down to only three issues per year (as above). If you want to go with twelve deadlines per year, they're necessarily going to overlap. You're going to have many more balls in the air, and papers won't move through the process in lock-step. Orchestrating this at scale will be an exciting challenge.

[parno]

Number of deadlines per year is definitely an interesting question, and it's quite helpful to hear about the JETS experience. Thanks! PETS started with 5 deadlines per year and ended up switching to 4, whereas VLDB started with and still has 12. Both reported that PC members have, on the whole, been quite good about returning reviews punctually (see also FAQ 11).

Yes, more deadlines will be more complex internally, especially for the chairs. However, the more frequently we have deadlines, the easier it is for researchers at all levels to make the decision not to submit early, and instead invest just a bit more time in producing something really amazing. As one example, in talking with people in the VLDB community, multiple professors commented on how nice it was to really concentrate on one student’s paper submission each month, rather than trying to prepare, say, five at once, which is the sort of benefit (and culture) that we'd really like to encourage in our community.

Of course, once we have some experience with the process, we can also consider adjusting the number of deadlines.

[warkdarrior]

Does having more deadlines incentivize authors to submit throughout the year? I can see authors would prefer to keep working on a paper and submit in late fall, with the hope of being accepted by the Feb 15 cutoff for conference publication. @danwallach, @parno: Did you observe such a trend with JETS or PETS, respectively? Or were submission loads evenly distributed throughout the year?

[parno]

It's a good question. If you look at the graph on slide 15 of this VLDB presentation http://www.vldb.org/2015/wp-content/uploads/2015/09/VLDB2015-PC-Presentation.pdf, it looks like there is a spike at the end, but about 73% of the papers are not submitted at the last minute. (Note that the graph is a bit confusing as it shows more than one year's worth, since it also shows the effects of resubmissions after the cutoff)

[danwallach]

We don't have enough long term data with JETS, which folded when USENIX decided it wasn't worth the resources. The PETS and especially VLDB folks should have a more detailed opinion on the load over time.

I will say that deadline management, from the program chairs' perspective, is a big deal. Each deadline is a bunch of work.

Dan

On May 18, 2017 3:58 PM, "Mihai Christodorescu" notifications@github.com wrote:

Does having more deadlines incentivize authors to submit throughout the year? I can see authors would prefer to keep working on a paper and submit in late fall, with the hope of being accepted by the Feb 15 cutoff for conference publication. @danwallach https://github.com/danwallach, @parno https://github.com/parno: Did you observe such a trend with JETS or PETS, respectively? Or where submission loads evenly distributed throughout the year?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ieee-security/ongoing-submission-plan/issues/9#issuecomment-302539464, or mute the thread https://github.com/notifications/unsubscribe-auth/AAtWddCE1p9vFo5ZgGDSlc0FScs8g8LOks5r7LDlgaJpZM4Jwzx7 .