search

acmsigsoft / artifact-evaluation

2 stars 4 forks source link

Artifact Automation #2

Open jon-bell opened 4 years ago

jon-bell commented 4 years ago

Should an artifact automatically generate any tables/graphs that appear in a paper, or is it OK (or, perhaps even better, given the potential for mistakes in these scripts) for this to be a somewhat manual process?

sbaltes commented 4 years ago

Regardless of whether the process is manual, semi-automatic, or automatic, it should be documented in a way that allows reviewers to understand and reproduce how the tables/graphs were created from the data to be able to verify their correctness.

piskachev commented 4 years ago

I think the "call for artifacts" should encourage the authors to make the steps automatic as much as possible. However, for manual steps, authors should be aware that reviewers should understand the artifact and judge its quality in a "reasonable" amount of time.