Scoring rubric for submissions review

[LouWoodley]

Rubric for scoring

• Alignment with the conference themes (1 - not at all 5 - very)

• Daily themes as well

• Appropriate spread of sessions across subthemes

• Relevance to the FORCE11 audiences (1 - not at all 3 - appeals to one audience? 5 - appeals to multiple audiences at FORCE11) - tag different potential audiences? [or: score to which audience(s)? to gauge spread BK]

  • Researchers
  • Publishers
  • Technologists
  • Funders
  • Librarians
  • Communicators?
  • Citizens / CitSci
  • Advocates / activists - OA, OS, etc.

• FORCE11 mission and goal alignment (spell out for reviewers - top topics) (1 - not at all 5 - very)

  • check overlap with fit with themes and subthemes - do we need this as separate criterion? [BK]

• Novelty of topic (1 - done to death 5 - top of mind right now)

• Creativity of format (1 - not at all 5 - very novel / interactive)

• Presenter qualification on topics - not a product pitch, balance of views, products ideas etc

• Potentially related submissions - List out other session numbers

• Speaker diversity - tag as newbie, regular etc

added from comments below:

• Speaker diversity - also M/F/X, geographical area, role
• Clarity of submission - is it clear what the presenter is planning to do during the session?

[bmkramer]

Re: speaker diversity: also M/F/X, geographical area, role

[bmkramer]

related to #101 Make review of submissions transparant

[EJ-Van]

Clarity of the submission. Is it clear what the presenter is planning to do during the session?

[bmkramer]

• I'd be happy to think of a way to organize scoring this in an online way based on the spreadsheet provided by Copernicus. We don't need to be limited by Excel :-)

• Lou suggested turning the above into a one-pager as instruction for reviewers

• After submission closes, can we make that scoring rubric public to promote transparancy?

• Other issues to discuss:

  • who will act as reviewers? (all programme committee members, excepting scoring self-submissions?)
  • early August: scan submissions for breadth and decide on promotion and/or inviting specific people/groups to submit?

[EJ-Van]

Hi everyone, Here is a first attempt at the submission rubric. I'm certain that we will have things to discuss and that some things will need to change, but it's a start. https://docs.google.com/spreadsheets/d/1HzvgZtbHDTkLHHr2qa0LFyyO4YuRn1DBjqB50I14amY/edit?usp=sharing

[EJ-Van]

Updates made based on feedback during the call last week. Talk to you all tomorrow.

[kamc]

How are we making this public?

[bmkramer]

I suggest we close this for now (together with #101), but make a note for next year.