Artefact Evaluation committee

[rolyp]

Expressions of Interest so far

See:

• Expression of Interest responses
• Reviewer volunteers collected at CI 2024

Summary Sentence

We need an AEC (Artefact Evaluation Committee) of ~16 members to allow for 2 reviews per submission and 2 reviews per reviewer.

See also:

• alan-turing-institute/climate-informatics-2024#3

To do

• [ ] @acocac Create draft and Expression of Interest form.
• [x] @rolyp Ask Eric Daub about who might be worth approaching in REG
• @rolyp Post request to REG Slack
• [x] @dorchard @MarionBWeinzierl Request to Cambridge/ICCS RSE teams?
• [ ] @acocac request to @ampersandmcd to post in CI social media, and repost using EDS book, EDS and ICCS accounts.
• [ ] @cassgvp to send invited to Turing E&S GC community channels
• [ ] Andrew Hyde to send to EDS Journal reviewers
• [ ] Invite CI submitted authors (full papers and abstracts)

Possible sources

• CI 2024 programme committee
• CI 2024 community
• Cambridge Research Software Engineering team
• Turing Research Software Engineering team

Benefits to Artefact Evaluation Committee members

• Reviewers will be listed as members of the Artefact Evaluation Committee on the Climate Informatics 2024 website.
• When the article addendum is published awarding a reproducibility badge to an article, reviewers will be credited with the Validation role of the CRedIT taxonomy.
• Opportunity to further develop reviewers’ reproducibility skills and best practices, with support from the CI2024 Reproducibility team.
• Reviewers will be acknowledged in the resulting documentation and workflow which will be adapted/reused for future Climate Informatics conferences.
• Cambridge University Press will provide a gift in the form of books to each reviewer, with the exact value depending upon the size of the Artefact Evaluation Committee.

---

We also considered (but ruled out for now) the publication of a final summary of their review feedback, as an “open review” accompanying the article. Environmental Data Science is transitioning to a transparent peer review model (all review reports published; reviewers have option of being named or remaining anonymous). We decided that an additional publication-like output for the review itself was probably unnecessary, but we do currently plan to allow reviewers and authors to (unanimously) opt-in to an open review process. (@dorchard made the case that a completely open process would probably not be desirable: it may make the reviewers feel as though they cannot be as fully critical as they might be in a blinded process.)

[cassgvp]

This is great, thanks @rolyp .

I've edited to add the CI community as source of AEC members, and added some more explicit benefits :).

Could you put numbers to how many people we might be able to recruit from each source? Have you connected with anyone from Turing REG already?

[rolyp]

@cassgvp I’ll check in with someone in REG today. (In particular I’m wondering whether identifying possible candidates first and approaching them individually might prove more fruitful than a blanket request for help sent to the entire team.) Eric Daub might be a good person to start with as I believe he is the Environment and Sustainability lead for REG so might have some ideas about who to approach.

[rolyp]

I don’t have any specific ideas about numbers so far..

[rolyp]

Unless anyone objects I’ll reply to Eric Daub (Turing REG) to say “provisionally September” as the likely timeframe for the AE process, so we can start recruiting members. Also thinking it might be good to aim for no more than ~2 Turing people and perhaps a similar number from ICCS so we don’t have too many people from a single organisation.