lessons learned

[timm]

note to all

none of the following are "definite" things to change... just notes for discussion to be shared with chairs of next conference's artifacts track

Items

• need some way to acknowledge candidates for best artifact as we go along. another label?
• need a tiger team
• need a requirements.md
• need separate advertisement rose and artifacts
• valuable doesn't mean runable (with caveats)
• set up the teams earlier
• need to review the labels (better guidelines)

[timm]

also

• invite to join goes to anonymous, need to let them they will receive. joint email

[timm]

need to debate what justifies replicated reporidced

• does not to be reported in a. peer reviewed setting (rude to industrial partners) but we need standards

[timm]

need to clarify that replicated and produced need evidence of available

[timm]

reviewers need tp nbe cleared on their current conclusions (trend them with " #" eg.

recommend available and repdicable

reject all badges

[timm]

need a mailing list for the authors and reviewers

• need to do broadcast to all

[minkull]

Authors should give an indication of the runtime (1 hour, 3 days?) and how to run a shorter version (eg 10 min) to check that it is functional.

[minkull]

Available requires a DOI or link to this repository along with a unique identifier for the object is provided.

Some are interpreting this as always requiring a DOI, but a link to a Git repo could also be seen as a unique identifier by others.

It would be good to clarify this point for future events, either by always requiring a DOI, or by specifying that certain web addresses ire also unique identifiers

[minkull]

README.md should start with the name of the paper that is applying for the available/reusable badges

[timm]

need a tight team of reviewers who do little initial reviewing but, just before deadline, jump in to handle emergency tasks (e.g. missing reviews)

[neilernst]

should consider file size limits. Some folks put out 6GB+ docker containers which in my view should be separated into data, scripts, libraries to be replicable.

[timm]

should consider file size limits. Some folks put out 6GB+ docker containers which in my view should be separated into data, scripts, libraries to be replicable.

we starting blocking those pull requests, allowing only .pdf (of the paper) and .md and img/* files that supported images in the .md.

[neilernst]

this wasn't the repo files in the Github for ROSE, but rather the referenced replication package.

I guess more broadly, what effort / possible bandwidth costs should the reviewer incur?

[ICSE20ArtifactAnonymousReviewer]

Not sure how (and whether) to address this: the interactive style of the issue discussions worked great and helped solving some bigger issues. However, I also see the risk of un-blinding reviewers as it is quite easy to guess where someone is, if you have a back and forth discussion over several hours or days...

[washizaki]

GitHub does NOT allow individual users to use multiple free machine accounts. Please refer to GitHub Terms of Service below. Assuming that most of PC members already had their own free machine accounts before ICSE Artifact review, most of PC members had to violate against the term of service by creating additional anonymous account for ICSE Artifact review. It would be better to have some explanation on this issue.

https://help.github.com/en/github/site-policy/github-terms-of-service#b-account-terms

[andivogelsang]

Expectations for reviewers should be clarified. Especially, are there any expectations or minimum requirements for the number of rebuttals? Are reviewers expected to review the third or fourth update from the submitters?