Closed danwallach closed 8 years ago
UlfarErl
commented
8 years ago Yes, the rate of reviewing is definitely a concern, and your other points are very useful as well. Thanks for all the detailed data! Bryan will have to ping in first here, with his toughts.
parno
commented
8 years ago Hi Dan,
Thanks for all of your thoughts. To simplify discussions and issue tracking, I'm going to close this Issue and open several more for each of the points you touched on.
-Bryan
We did a VLDB-ification when moving the USENIX EVT/WOTE workshop to become JETS. Here are some thoughts and issues that came up for us.
Conference vs. journal. For lack of a better term, what S&P is building is a hybrid or "journfrence". While most university promotion & tenure committees have had this distinction in CS beaten out of them, it's technically feasible to label the resulting publications as "journal" publications rather than "conference". This has essentially no downside (maybe you have to reformat your paper in the single-page "journal" style) and lots of upside (looks good on the vita). Each point at which you release the newly accepted papers corresponds to an "issue" of the journal, and the set of all the papers to appear in any given year is a "volume". I'd encourage S&P to go with the "journal" aspect.
Number of deadlines per year. We initially scheduled JETS with four submission deadlines. This made things remarkably tight so we shifted to three, which felt much more reasonable. Here's the schedule that the USENIX production staff and we ultimately converged on:
The gist of this was that we had three deadlines, wherein we assigned reviewers and went through the usual reviewing process. A month later, we'd have the first round of decisions, which had one important gate: either you're in (done!), you need to make minor revisions (due in a month), or you need to make major revisions (and you would then resubmit to the next deadline). In practice, the question of "do you think the authors can address your concerns in one month?" served to help resolve which path was appropriate for each paper.
Also, this process played relatively nicely with HotCRP, wherein we had a new HotCRP instance three times per year. Eddie Kohler added a feature for me that made it easy to dump the reviewer database (including area preferences) and load it into subsequent HotCRP instances. HotCRP otherwise isn't set up to manage the sort of dataflow typical of an open-ended journal (which may or may not be an issue for S&P).
Keeping reviewers on task and hitting deadlines. I had to do a fair bit of browbeating to ensure that we hit all these deadlines, which is part of why we simplified down to only three issues per year (as above). If you want to go with twelve deadlines per year, they're necessarily going to overlap. You're going to have many more balls in the air, and papers won't move through the process in lock-step. Orchestrating this at scale will be an exciting challenge.
On-boarding and off-boarding the editorial board members. There were some members of our board who were fantastic about getting their assignments done. Others? Not so much. I don't have fantastic advice about how to get people engaged and keep them productive, nor can I say when it's time to cut the cord. My advice is to create multiple tiers of engagement, and allow people to ramp up and down their commitments. It's probably a good idea to be super inclusive for the editorial board, letting you scale the number of reviews per reviewer without killing anybody. You might also want to consider subcommittees on particular hot topics for any given year, so if we get 30 papers on the same topic which get disjoint reviewers, the subcommittee members will be likely to have better coverage and can reduce some of the randomness inherent in the review process.