This guide details how the PyCon Program Committee operates. It's designed to be a working document for the PC itself to follow, and also for prospective PC members to get an idea of what being on the PC entails.
This is a living document; it describes the PyCon process as it is "right now"; it'll be updated continuously as our process grows and changes.
The program committee's mission is to craft the best possible program for PyCon out of the pool of proposals submitted. It's a tough job: PyCon 2013 received roughly 600 proposals for only 110 slots. The committee tries to go beyond simple review: committee members interact with proposal authors and try to help them put forward the best possible proposals. Often the final form a talk takes is shaped by feedback, comments, and criticism from the committee.
Joining the committee
Committee membership is open to anyone - more voices, more diversity of viewpoints, make for a better conference. To join the program committee, sign up for the pycon-pc mailing list. We use the mailing list to coordinate all the rest of our activities.
Committee membership is a time commitment. Expect to spend something approaching 4-8 hours per week, August through December, with the bulk of that work coming towards the end of November as the final selections are made. The bulk of the work involves reviewing proposals on the PyCon website (see initial review,
below) and then later participating in IRC meetings (Kittendome and Thunderdome, below).
Committee structure, consensus, and voting
The committee's led by a chair and co-chair. In 2014, the chair is Luke Sneeringer; the co-chair is
Alex Gaynor.
Whenever possible we strive to operate in a way drawn from the open source process that's worked so well for Python. That means that we'll try to find rough consensus where we can. This doesn't scale for review itself, so during the review process, we'll use voting and majority rule to select the bulk of the talks.
Ultimately, the responsibility for the program falls on the chair, so the chair has the final say. The chair is the Benevolent Dictator for the PyCon Program, if you will. However, the chair will try to use chair fiat as little as possible.
The review process
After someone submits a talk proposal, it goes through three rounds of review:
Initial review takes place on the website. The committee reviews each proposal, leaves feedback for the author, and votes on the proposal using an "identify the champion" system. We also meet occasionally on IRC to discuss talks and coordinate feedback to presenters.
Once the call for proposals closes, IRC review begins. The first round, nicknamed "kittendome", reviews each talk in isolation, determining if the proposal would be a "good talk" for PyCon. Some talks will be rejected at this point; the rest will pass on to…
The second round, nicknamed "Thunderdome". The remaining talks (typically about half of the proposals) will be grouped by topic into groups of 3-5. During Thunderdome, we'll discuss one group at a time, and select the best talk(s) from each group.
When Thunderdome completes, we'll have the list of selected talks. Authors will be notified of acceptance or rejection, the selected talks will be published, and the conference schedule will be written.
2014 Timeline
This is a rough idea of the schedule we're looking at:
Website reviews: July 1, 2013 – September 15, 2013
Kittendome meetings: August 15, 2013 – November 1, 2013
Thunderdome meetings: November 4, 2013 – November 20, 2013
Talk list published: December 1, 2013
Schedule published: December 15, 2013
Initial review
Once talks start to roll in, we'll review them using tools on the website. This happens concurrently with the CFP - we don't wait for the call to close to start this work. The committee has two goals here:
Give feedback to authors and help them improve their proposals (authors can
edit proposals up until the CfP closes).
Get an initial set of votes from as many committee members as possible. This
will inform the later meetings.
We try to make sure that each talk gets reviewed by at least 3 committee
members (and ideally more).
Voting
The process we use to assess proposals is based on the Identify the
Champion process. The idea is to to
assess each talk and identify "champions" and/or "anti-champions" -- people who
will argue strongly for or strongly against a talk in initial review.
The document linked above uses an "A/B/C/D" scale for reviewing; we use the
same idea, but use "+1/+0/-0/-1" labels (after the Apache voting system).
These votes mean:
+1 -- I will champion the talk at the program committee meeting. The
topic sounds good and the proposal is solid. This vote means you're willing
to put your reputation on the line in favor of the proposal and its author,
and that you believe PyCon would be worse off for not having the talk.
+0 -- This topic is good and the proposal is solid. The speaker is
capable of correcting any deficiencies, and I think a significant number of
PyCon attendees would want it available. I don't feel strongly enough about
it to serve as its champion at the program committee meeting.
-0 -- I am not in favor of this talk as proposed. The talk, the topic,
or the proposal has problems that concern me. I'd rather see it not
accepted.
-1 -- This talk or its proposal has significant problems. I believe that
PyCon would be worse off for having it, so I will argue strongly to reject
this talk.
If you don't know enough about a topic to judge a proposal's coverage of it, or
it's a topic you tend to actively avoid, you should not recuse yourself from
voting on that basis alone. You can still judge the structure of the proposal,
the likelihood that the speaker can cover the material proposed in the time
allotted, whether the topic is likely to appear to others, etc.
Criteria for evaluating talks
It's hard to come up with an objective set of criteria for evaluating a talk.
Ultimately, scoring talks is a subjective endeavor; we're all trying to select
what we think are the "best" talks for the conference. We don't expect everyone
to use the same set of criteria, but some valuable questions you might ask
yourself as you review are:
Would I go to see this talk?
Are there a significant group of likely attendees that would want to see this talk?
Does the proposal seem well-thought-out?
Does it seem like the author knows enough about the topic?
Is the proposal too ambitious (or too paltry) for the time allotted?
"Obvious" accepted/rejected talks
After the initial review, there will be a small set of talks that are "obviously
good" or "obviously bad". We'll set a threshold, by committee consensus, for
talks that are automatically rejected or get to automatically skip ahead to
Thunderdome. This is usually something like "reject all talks with 3 or more -1s
and no +0s or better" or "accept all talks with at least 4 +1s and no -1s".
These accepted talks aren't a free pass to PyCon - it just means that the talk
goes directly to Thunderdome. A reject is final, however; it weeds out the
few obviously-bad proposals that nobody on the PC wants to support.
IRC Meetings
Next, we hold meetings in IRC. (The committee chairs can help any committee
members who aren't familiar with IRC.) Meetings run for an hour, and are usually
scheduled a week or two in advance.
Program Committee meetings are held in IRC on Freenode. If you're not familiar
with IRC, ask Luke or Alex and we can help you get set up.
We'll announce each meeting's itinerary in advance on the PC list, including the
list of talks to be covered.
Kittendome
In the first round of meetings, we go through each proposal individually. The
here is simple to determine if a given talk -- reviewed in isolation -- is
potentially good enough for PyCon. That is, in the first round we review talks
strictly on their merits and avoid holding them up against similar talks
(we do that next, in Thunderdome_).
We'll review talks, one at a time. We'll give a couple minutes for the champion
(identified by the website votes) to make an argument for the talk. Then,
a few minutes of general debate. Finally, we'll vote yay/nay on each talk.
Majority rules - a majority of "yay" votes will indicate that the talk is
accepted -- it moves onto Thunderdome. A majority of "nay" votes indicates
that the talk is rejected -- it's out. The chair will break ties.
Meeting format
Meetings will be in IRC; we use an IRC 'bot to keep us on track. A Kittendome
meeting will look like this:
First, the bot will announce the talk is under consideration (note that
the format has changed slightly since 2012, but the principle remains the
same):
<pycon_bot> === Talk 456: Monitoring parrot liveliness - http://us.pycon.org/2012/review/456/ ===
<pycon_bot> (http://us.pycon.org/2012/review/123/ will be next)
<pycon_bot> If you are (a/the) champion for #456, or willing to champion it,
please type a succinct argument for inclusion of this talk. (2 Minutes).
Say 'done' when you are finished.
A champion — usually someone who's given the talk a "+1" on the website -- then
has 2 minutes to give a succinct argument for the talk. Many champions prepare
these summaries in advance (the schedule goes out a day or two prior to the
meeting):
<champion> Love the approach. Very clear what the talk will accomplish, and
how, and in a valuable topic. done.
The bot then announces general debate:
<pycon_bot> === General Debate (3 minutes) for Talk: #456 ===
Members debate, and anything goes at this point (but please see the meeting
guidlines below).
Once debate concludes, it's voting time:
<pycon_bot> === Voting time! yay/nay votes for talk #24 ===
<pycon_bot> Please do not speak after voting until we've gotten our report.
<champion> yay
<otherperson> nay
<someoneelse> yay
After a short voting period, the bot summarizes the results:
<pycon_bot> === Talk Votes on #456: 3 yays, 10 nays, 0 abstentions ===
<pycon_bot> The nays have it.
The bot records the final decision. Rinse and repeat.
Thunderdome
After round one has ended, we're left with a set of talks that are all "good
enough" for PyCon. Unfortunately, this'll typically be more than double the
number of talks we can actually accept, so we pit similar talks head-to-head
and try to select the best.
In each "round" of Thunderdome we review 3-5 talks, roughly grouped by
theme/content/topic/etc. Out of each group we'll get one or more winners,
possibly some damaged talks, and some talks that are out. We'll do this
by discussing, and then the committee will vote for the talks they like.
Members may vote for any number of talks (including none or all of them).
The winners are the talks from the group that absolutely should be at
PyCon. It advances and is safe from future cuts. It's on the program! There
doesn't have to be a winner from each group, but there probably should be. The
winners are any talks that have a 75% supermajority.
Damaged talks are ones that have been hurt by Thunderdome but aren't quite
out of the running yet. They'll be put into the damaged pool and if at the end
we've still got spaces left we'll fill those spaces from the damaged pool. There
may not be any damaged talks in a given group, or all talks may end up damaged.
Damaged talks are are any talks that didn't win, but did receive votes from at
least 50% of the committee.
Talks are selected by fewer than 50% of the committee in attendance are out
– they will not appear at PyCon.
The result of Thunderdome will be the ~100 talks that'll appear on the PyCon
program.
Thunderdome process
Like the first round we'll use a bot to keep us on track.
The bot will announce each group:
<pycon_bot> === Thunderdome for “Birds of a feather” begins now! ===
<pycon_bot> Talk #123: Modelling flight patterns of swallows
<pycon_bot> Talk #456: Monitoring parrot liveliness
<pycon_bot> Talk #789: SciPy for aerodynamics engineers
<pycon_bot> You now have 2 minutes to review these talks and collect
your thoughts prior to debate. Please refrain from speaking until
debate begins.
Members will have 2 minutes to read talk descriptions, collect their thoughts,
and formulate arguments; please try to stay silent during this time so everyone
can think.
The bot will then announce 5 minutes of open debate:
<pycon_bot> === General debate (5 min) for “Birds of a feather” ===
Members debate, and anything goes at this point (but please see the meeting
guidlines below).
Then we vote! Vote for as many talks as you like, but please keep in mind the
very limited number of spots. Voting will look like this:
<pycon_bot> === Voting time! ===
<pycon_bot> Enter your vote in the form “123, 456, 789”. You may vote for as many talks as you like, but please keep in mind the limited number of available slots.
<jacob> 123, 456
<jesse> 456
<tim> 456
…
The bot will tally the scores, telling us the results:
Remember there may be no winner, and there may be any number of damaged talks.
Finally, the chair will record the talks as in, damaged, or out, based on the
voting.
Meeting guidelines
To keep things moving smoothly and quickly, please try to follow these guidelines
during all IRC meetings:
The chair will send an agenda for each meeting to the mailing list in advance.
This'll list all the talks we plan to discuss. If you're a champion for
one of the talks up for that meeting, please try extra-hard to make it
to the meeting! Otherwise we'll have to "channel" you based on what you
wrote on the website, and that won't do your voice justice.
Please use the time before each meeting to refresh your memory on the talks
for each meeting; you'll only have a short period of time during the meeting
to read each talk. Please come prepared.
Please stay on topic. We have a lot to do, and not much time to do it. Please
direct any unrelated chatter to private chat, other chat rooms, or the mailing
list.
For Kittendome, focus only on the talk itself, not on other talks. If there's
another, better talk on the same topic try to ignore it; we'll sort it out in
Thunderdome. In this round, just deal with each talk on its (de-)merits.
In Thunderdome, focus on the talks in each group, and not on other groups,
other talks that didn't make it to Thunderdome, talks you wish had been
submitted, etc. It's impossible to evaluate every talk against every other
talk, so we have to work in groups.
Disclose bias if it exists. If the speaker is a friend, colleague, coworker,
competitor or nemesis that's completely okay, but tell the group. If you wrote
a piece of software covered by a talk, say so. Bias, in and of itself,
isn't really an issue -- we have a big enough pool of reviewers that it'll
balance out. But if you are biased, tell us so that we know where your
review is coming from.
No talks will be tabled once the meeting begins. Thus, if you feel strongly
about a group, topic, or talk then make a special effort to come to the
meeting! If you absolutely can't make it, email the chair and a talk can be
moved to a different meeting, but please let the chair know well in advance.
Generally, debate will not be extended. Again, we have too much to get through
to spend extra time. However, if there's a particular contentious debate,
the chair may choose to extend debate, so ask the chair if you think the group
deserves extra time.
The chair gets the final call. Most of the time the votes will identify a
clear winner, and the chair should follow the votes 99% of the time. However,
if there's a tie, or if votes are really close, or if the voting is otherwise
not quite clear, the chair gets to make a judgement call. This should
happen rarely, but it will happen.
Decisions are final. Once a decision has been made, discussion should end and
we'll move on. If you feel a decision was made in error, bring it up on the
mailing list or via private email to the chair.
If a talk under discussion has been submitted by you, you'll need to recuse
yourself. Do this by leaving the chat room; the chair will invite you back
when discussion is over.
This guide details how the PyCon Program Committee operates. It's designed to be a working document for the PC itself to follow, and also for prospective PC members to get an idea of what being on the PC entails.
This is a living document; it describes the PyCon process as it is "right now"; it'll be updated continuously as our process grows and changes.
The program committee's mission is to craft the best possible program for PyCon out of the pool of proposals submitted. It's a tough job: PyCon 2013 received roughly 600 proposals for only 110 slots. The committee tries to go beyond simple review: committee members interact with proposal authors and try to help them put forward the best possible proposals. Often the final form a talk takes is shaped by feedback, comments, and criticism from the committee.
Joining the committee
Committee membership is open to anyone - more voices, more diversity of viewpoints, make for a better conference. To join the program committee, sign up for the pycon-pc mailing list. We use the mailing list to coordinate all the rest of our activities.
Committee membership is a time commitment. Expect to spend something approaching 4-8 hours per week, August through December, with the bulk of that work coming towards the end of November as the final selections are made. The bulk of the work involves reviewing proposals on the PyCon website (see initial review, below) and then later participating in IRC meetings (Kittendome and Thunderdome, below).
Committee structure, consensus, and voting
The committee's led by a chair and co-chair. In 2014, the chair is Luke Sneeringer; the co-chair is Alex Gaynor.
Whenever possible we strive to operate in a way drawn from the open source process that's worked so well for Python. That means that we'll try to find rough consensus where we can. This doesn't scale for review itself, so during the review process, we'll use voting and majority rule to select the bulk of the talks.
Ultimately, the responsibility for the program falls on the chair, so the chair has the final say. The chair is the Benevolent Dictator for the PyCon Program, if you will. However, the chair will try to use chair fiat as little as possible.
The review process
After someone submits a talk proposal, it goes through three rounds of review:
When Thunderdome completes, we'll have the list of selected talks. Authors will be notified of acceptance or rejection, the selected talks will be published, and the conference schedule will be written.
2014 Timeline
This is a rough idea of the schedule we're looking at:
Initial review
Once talks start to roll in, we'll review them using tools on the website. This happens concurrently with the CFP - we don't wait for the call to close to start this work. The committee has two goals here:
We try to make sure that each talk gets reviewed by at least 3 committee members (and ideally more).
Voting
The process we use to assess proposals is based on the Identify the Champion process. The idea is to to assess each talk and identify "champions" and/or "anti-champions" -- people who will argue strongly for or strongly against a talk in initial review.
The document linked above uses an "A/B/C/D" scale for reviewing; we use the same idea, but use "+1/+0/-0/-1" labels (after the Apache voting system). These votes mean:
If you don't know enough about a topic to judge a proposal's coverage of it, or it's a topic you tend to actively avoid, you should not recuse yourself from voting on that basis alone. You can still judge the structure of the proposal, the likelihood that the speaker can cover the material proposed in the time allotted, whether the topic is likely to appear to others, etc.
Criteria for evaluating talks
It's hard to come up with an objective set of criteria for evaluating a talk. Ultimately, scoring talks is a subjective endeavor; we're all trying to select what we think are the "best" talks for the conference. We don't expect everyone to use the same set of criteria, but some valuable questions you might ask yourself as you review are:
"Obvious" accepted/rejected talks
After the initial review, there will be a small set of talks that are "obviously good" or "obviously bad". We'll set a threshold, by committee consensus, for talks that are automatically rejected or get to automatically skip ahead to Thunderdome. This is usually something like "reject all talks with 3 or more -1s and no +0s or better" or "accept all talks with at least 4 +1s and no -1s".
These accepted talks aren't a free pass to PyCon - it just means that the talk goes directly to Thunderdome. A reject is final, however; it weeds out the few obviously-bad proposals that nobody on the PC wants to support.
IRC Meetings
Next, we hold meetings in IRC. (The committee chairs can help any committee members who aren't familiar with IRC.) Meetings run for an hour, and are usually scheduled a week or two in advance.
Program Committee meetings are held in IRC on Freenode. If you're not familiar with IRC, ask Luke or Alex and we can help you get set up.
We'll announce each meeting's itinerary in advance on the PC list, including the list of talks to be covered.
Kittendome
In the first round of meetings, we go through each proposal individually. The here is simple to determine if a given talk -- reviewed in isolation -- is potentially good enough for PyCon. That is, in the first round we review talks strictly on their merits and avoid holding them up against similar talks (we do that next, in Thunderdome_).
We'll review talks, one at a time. We'll give a couple minutes for the champion (identified by the website votes) to make an argument for the talk. Then, a few minutes of general debate. Finally, we'll vote yay/nay on each talk. Majority rules - a majority of "yay" votes will indicate that the talk is accepted -- it moves onto Thunderdome. A majority of "nay" votes indicates that the talk is rejected -- it's out. The chair will break ties.
Meeting format
Meetings will be in IRC; we use an IRC 'bot to keep us on track. A Kittendome meeting will look like this:
First, the bot will announce the talk is under consideration (note that the format has changed slightly since 2012, but the principle remains the same):
A champion — usually someone who's given the talk a "+1" on the website -- then has 2 minutes to give a succinct argument for the talk. Many champions prepare these summaries in advance (the schedule goes out a day or two prior to the meeting):
The bot then announces general debate:
Members debate, and anything goes at this point (but please see the meeting guidlines below).
Once debate concludes, it's voting time:
After a short voting period, the bot summarizes the results:
The bot records the final decision. Rinse and repeat.
Thunderdome
After round one has ended, we're left with a set of talks that are all "good enough" for PyCon. Unfortunately, this'll typically be more than double the number of talks we can actually accept, so we pit similar talks head-to-head and try to select the best.
In each "round" of Thunderdome we review 3-5 talks, roughly grouped by theme/content/topic/etc. Out of each group we'll get one or more winners, possibly some damaged talks, and some talks that are out. We'll do this by discussing, and then the committee will vote for the talks they like. Members may vote for any number of talks (including none or all of them).
The winners are the talks from the group that absolutely should be at PyCon. It advances and is safe from future cuts. It's on the program! There doesn't have to be a winner from each group, but there probably should be. The winners are any talks that have a 75% supermajority.
Damaged talks are ones that have been hurt by Thunderdome but aren't quite out of the running yet. They'll be put into the damaged pool and if at the end we've still got spaces left we'll fill those spaces from the damaged pool. There may not be any damaged talks in a given group, or all talks may end up damaged. Damaged talks are are any talks that didn't win, but did receive votes from at least 50% of the committee.
Talks are selected by fewer than 50% of the committee in attendance are out – they will not appear at PyCon.
The result of Thunderdome will be the ~100 talks that'll appear on the PyCon program.
Thunderdome process
Like the first round we'll use a bot to keep us on track.
The bot will announce each group:
Members will have 2 minutes to read talk descriptions, collect their thoughts, and formulate arguments; please try to stay silent during this time so everyone can think.
The bot will then announce 5 minutes of open debate:
Members debate, and anything goes at this point (but please see the meeting guidlines below).
Then we vote! Vote for as many talks as you like, but please keep in mind the very limited number of spots. Voting will look like this:
The bot will tally the scores, telling us the results:
Remember there may be no winner, and there may be any number of damaged talks.
Finally, the chair will record the talks as in, damaged, or out, based on the voting.
Meeting guidelines
To keep things moving smoothly and quickly, please try to follow these guidelines during all IRC meetings:
(last updated: June 24, 2013)