Artefact Evaluation process

[rolyp]

Summary Sentence

A description (see below) of the process/workflow we will use for CI 2024 artefact evaluation.

See also:

• alan-turing-institute/climate-informatics-2024#12
• alan-turing-institute/climate-informatics-2024-ae#3
• alan-turing-institute/climate-informatics-2024-ae#2
• alan-turing-institute/climate-informatics-2024-ae#4

• 6

• 7

---

See working document.

---

Remaining tasks

• [ ] Resolve whether we need the "Versioning" subsection
• [ ] Clarify that authors won't be anonymous (?)
• [ ] Guidelines for reviewers, including:
  • [ ] Expected HotCRP workflow (see #4 for details)
  • [ ] Reviewers can opt to deanonymise?
• [ ] Details of retrospective report
• [ ] Clean up any last details and remove any ==comments==
• [ ] Work out where to publish this and how to circulate to the authors

[rolyp]

@dorchard @acocac @geo-rao @cassgvp Imported from Dominic’s HackMD with some further edits from me. For now I’ve put this text in the issue itself, but @cassgvp it might be worth putting this into a (versioned) file instead (if you can suggest a good place to put it in the repo).

[cassgvp]

@rolyp , I think we probably need an "artefact-review" directory in the root. There will be a lot of docs and stuff we'll need to manage.

Alternatively, maybe it should all happen in a whole new repo (and we move the relevant issues)? Would be nice to keep this clean and transferable to future AE review efforts. What do you think?

[rolyp]

@cassgvp Yes I’d be inclined towards a separate repo, to keep this one specific to the website and reduce the risk that an update triggers a Git workflow that involves the website. Maybe climate-informatics-2024-ae (bit of a mouthful but at least makes the relationship clear!) We can migrate the issues.

[cassgvp]

Awesome. I've set up https://github.com/alan-turing-institute/climate-informatics-2024-ae and added @rolyp @acocac @MarionBWeinzierl @dorchard as admins :).

[dorchard]

I have just written a bunch of instructions on what reviewers should provide (What authors should provide in their artefact submission). Comments welcome! Perhaps we can move this over to a file now so it's easier to track our contributions?

[rolyp]

@dorchard Extracted to process.md.

[dorchard]

Thanks. Maybe we can turn the issues then into proper issues with checklists of what we think is remaining then?

[dorchard]

Okay remaining tasks list is in the issue now.