ieee-security / ongoing-submission-refinement

Public forum for discussing refinements to IEEE S&P's ongoing submission model
Creative Commons Zero v1.0 Universal
9 stars 0 forks source link

We should carefully consider which month to start the quarterlies on #3

Open zardus opened 5 years ago

zardus commented 5 years ago

This is heavily related to #1, but from the deadline perspective. I don't want to hijack #1, so I opened up this issue.

I understand why March is currently set as the start of the cycle (the last deadline being in February), but according to my calculations, this seems to be the worst cycle, and a cycle starting in January is best. Here is the code I used (it requires Python 3.6 at a minimum because of the use of f-strings):

import datetime
import calendar

def day_delta(month,date): return abs((datetime.date(year=2020, month=month, day=1)-date).days)

deadlines = {
    datetime.date(year=2020, month=2, day=5): 'CCS', # 2020 CFP not yet up; estimate
    datetime.date(year=2020, month=2, day=15): 'Usenix',
    datetime.date(year=2020, month=5, day=15): 'Usenix',
    datetime.date(year=2020, month=5, day=16): 'CCS', # set to 16 for key uniqueness; actually 15
    datetime.date(year=2020, month=6, day=14): 'NDSS',
    datetime.date(year=2020, month=8, day=23): 'Usenix',
    datetime.date(year=2020, month=9, day=13): 'NDSS',
    datetime.date(year=2020, month=11, day=15): 'Usenix',
}
assert list(deadlines.values()).count("CCS") == 2
assert list(deadlines.values()).count("NDSS") == 2
assert list(deadlines.values()).count("Usenix") == 4

oakland_schedules= [
    [ m for m in range(m_start, 13, 3) ]
    for m_start in range(1, 4)
]

closest_deadlines = [
    {
        oakland_month: sorted(deadlines.items(), key=lambda other_deadline: day_delta(oakland_month, other_deadline[0]))[0]
        for oakland_month in oakland_rotation
    }
    for oakland_rotation in oakland_schedules
]

distances = {
    min(cd_option.keys()):
    [ day_delta(oakland_month, closest_deadline[0]) for oakland_month,closest_deadline in cd_option.items() ]
    for cd_option in closest_deadlines
}

distances_sum = { k:sum(v) for k,v in distances.items() }
distances_min= { k:sum(v) for k,v in distances.items() }

for k,v in distances.items():
    print(f"If we start quartery deadlines in {calendar.month_name[k]}, we will have a total of {sum(v)} and a minimum of {min(v)} days around Oakland deadlines.")

This code is a bit chaotic; sorry about that.

The output is:

If we start quartery deadlines in January, we will have a total of 114 and a minimum of 17 days around Oakland deadlines.
If we start quartery deadlines in February, we will have a total of 54 and a minimum of 4 days around Oakland deadlines.
If we start quartery deadlines in March, we will have a total of 53 and a minimum of 9 days around Oakland deadlines.

So a cycle starting in January gives us more than twice as many "non-deadline" days around Oakland deadlines in total (114 days, as opposed to 53 days for the march cycle) and almost twice as many in minimum (17 days for January, compared to 9 for March). In terms of spreading out deadlines with the other top4, starting in January is best. I think this would be also optimal for providing more flexibility in choosing the right deadline and submitting a paper when it is ready (as stated in the observed advantages), but of course it's not a huge deal either way.

My guess (and hence, the relationship to #1), is that optimally spreading out deadlines will also somewhat help with spreading out reviewing load. As that discussion mentions, it's hard to have an impact on deadlines of other conferences, but we can position Oakland deadlines in optimal months.

If we want to do that, the best way is probably to start in April, as then we'll have a February deadline to cap off monthlies for Oakland 2020, and going forward, the final submission for possible publication in the next Oakland would be January.

Finally, I'm totally happy to be wrong about all this. Just wanted to bring it up in case it's useful!

ThorstenHolz commented 5 years ago

Thanks a lot for this feedback Yan! Actually the dates were determined by the last submission cycle given that we have some time restrictions from the publisher for the proceedings. So we started with the last cycle (decisions due middle of February) and counted backwards, which led to the outlined submission dates. We still need to check with the publisher how much wiggle room we have (after all, we do not print anymore but have electronic proceedings), your script certainly shows that March 1 is not the optimal date (as also discussed in #1). We definitely need a better synchronization between the major conferences, but this is easier said than done...