Add feedback to rejection emails

acuppy commented 7 years ago

Having been on the review committee for RailsConf 2016, and been a submitter for many before it, I think constructive feedback in the rejection email would alleviate the anxiety of the unknown and could encourage speakers to submit again in the future.

I spoke with an individual last year at RailsConf 2016 that didn't know why his proposal was rejected; He didn't know if it was totally off or close. I logged in, and told him he was close, but it just didn't make it for the year. Due to that he resubmitted the talk this year and was accepted. He told me that he might not have, had he not known it was close.

I can speak from experience: I've been rejected, but not sure if it was due to something I could address in the future, or it simply didn't win out over other talks. You're left "in the dark."

Whatever solution is found, I feel it needs to:

Have little impact on the review committee's process
Provide constructive and direct feedback
Be encouraging and supportive and not callus and blunt
Close the loop on the process and not leave a vague sense that the door is still open for the current event

After speaking with @mghaught it appears that many/most talks fall into one of a few buckets:

Talk lacks clear attendee value
Talk is vague or too broad
Talk topic is overdone
Talk topic was out of alignment with the goals of the event
Talk was a runner up, but another talk won out
(likely a few more)

As we all know, putting a talk together can be time consuming and scary. Given feedback can make the experience encouraging and empowering. That's the ultimate goal.

(I'll include implementation thoughts below)

mghaught commented 5 years ago

I'd love some concrete ideas on how this could work. Up to this point, the closest we've come to this is to encourage speakers to reach out for feedback and one by one the program committee responds as they have time.

acuppy commented 5 years ago

An idea...

Organizer criteria & suggestions

Organizers could establish criteria for reviewers that they checked as they reviewed each proposal ("Does this focus on the attendee?" "Do you feel the talk would fit the session timeline?" "Does the talk have a good narrative?" etc.). The reviewer could check each like a survey. If the list is incomplete, it could prompt the reviewer with a prefilled reply to the submitter (ex: "Can you clarify in your proposal how the attendee would benefit?"). Organizers could write the criteria question and the attendee feedback question. The submission criteria could be presented to the submitter when they fill out the form. If nothing else went to the reviewer but the preformatted feedback to the submitter, it starts to bring more value to their experience.

As an alternative to the prefilled email, the criteria could be displayed with the proposal to the submitter as it moves through the feedback process...

Criteria	Review 1	Review 2	Review ...
Clear title	X	X	X
Clear audience	-	-	-
Aligned with conference goals	X	-	X
...

The visuals might be different, but aligning reviewers and submitters is a good thing. Sure, there will be disputes, but that's true already. The difference is that there's little exposure to the process on the submitter's behalf.

Alternatively, this could be a summary and not a history...

Criteria	Feedback
Clear title	Agree
Clear audience	Disagree
Aligned with conference goals	Agree
...

or a combination of the 2+

Review status

I feel there's no harm in letting submitters known that their talk is progressing through the review process: "Awaiting Review," "Reviewed," "Under Further Consideration," "Accepted" and "Not Accepted" (not "rejected"). These can be automated as the first review and subsequent reviews are done. If nothing else, it gives a sense that a human reviewed the talk more than just the title.

Email Feedback

In the case where a talk is not accepted, this feedback could be combined into a prefilled email...

If it's a form field of markdown, it could be edited before sent. I understand there are a lot of rejections, and doing this one at a time may be too much, so that's where prefilled bullets come in.

Hi Sarah!

Thank you for your submission. First and foremost, we're honored you put the time and energy into submitting this year. Unforunately, your talk {{talk title}} was not accepted for this years program. There were over {{total talks}} this year, and the review committee could only select {{accepted talks}}. 

Here are a few thoughts our review committee had on your submission:

{{bullet point list of what criteria they hit, phrased in supportive language}}

And, some constructive critism for you to consider next time:

{{bullet point list of what criteria they missed, phrased in supportive language}}

We understand that you may have invested a lot of your own time into writing your proposal, and that a "not accepted" answer can be discouraging. However, please don't stop here. Submit more talks to events like this one.

We look forward to hearing from you next event!

- {{event}} Review Committee

Hope that helps :)

rubycentral / cfp-app