alan-turing-institute / climate-informatics-2024-ae

Artefact evaluation (AE) of research objects submitted to Climate Informatics 2024

Other

2 stars 0 forks source link

Artefact Evaluation platform #4

Open acocac opened 3 months ago

acocac commented 3 months ago

Summary Sentence

A platform to handle the artefact evaluation process.

What needs to be done?

[x] Conduct pre-feasibility assessments of suitable management platforms.
[ ] Create mini-tutorials how to use the selected platform.
[ ] Set-up the platform for the AE process.

See also:

alan-turing-institute/climate-informatics-2024-ae#1

Platforms

HotCRP

https://hotcrp.com
Test sites created by @acocac and @rolyp

System admin (to set before)

Users
Topics
Deadline
Review Scores
- Overall merit
- Reviewer familiarity
Labels for badges

Users

Add accounts ok, but
- require user to sign in manually
- invite in the SPAM folder (Alejandro's gmail account)
- Problems with inviting existing accounts (Roly)
Manually confirm with program committe

Submission (Authors)

Do we need separate author and reviewer checklists? (Probably just latter.)
Checklist
- Add README or artifact-evaluation.md (example here)
- Create a branch

Review

Initial dummy review to enable interaction with authors (15 days)
- PC chair notifies reviewers of a hard deadline to submit a draft reviews and began discussion about what badges to award
Actual review comes towards the end of the process (after author interaction) (5 days)
Authors submit revised artifact (ideally) by pushing a new revision to repo (or updating the existing one)
Review Scores
- Overall merit: available, functional, reusable
- Reviewer expertise: some familiarity

Badges

Not in HotCRP
Visible in the published paper
Artefact (version assessed) curated in Zenodo, doi

rolyp commented 3 months ago

@acocac Renamed to avoid any suggestion that the platforms are used to manage the artefacts themselves!