timm
commented
3 years ago @klainfo did updates. please check https://github.com/researchart/patterns/compare/8a270b3..da2e785
reopen this issue if i got it wrong
Closed klainfo closed 3 years ago
timm
commented
3 years ago @klainfo did updates. please check https://github.com/researchart/patterns/compare/8a270b3..da2e785
reopen this issue if i got it wrong
Hi Tim,
Thanks for the initiatives! I really like the artifact standards. https://github.com/researchart/patterns/blob/master/standards/artifact.md
Below, I added some common issues when I co-chaired the ROSE@ICSME'20 that I received from reviewers?
An artifact requires many dependencies (java, windows, ...) but reviewers do not have appropriate environments to evaluate (e.g., can't run an exe file, can't run a jar file, too many library dependencies) so how can we ensure that the artifact submission is cross-platform. Should we add technical expertise (e.g., Windows or Linux, java or python or R) when bidding artifacts, in addition to the domain expertise?
For a python package, I believe requirements.txt is recommended.
An artifact takes forever to run so how can we ensure that an artifact can be reproducible in a manageable amount of time. Should we add a max runtime (e.g., 15 mins)? Anything more than that requires a smaller size of data?
We need an example README template as well. I recommend (this one).
Some highly recommended resources https://guides.lib.berkeley.edu/reproducibility-guide.
Kind regards,
Kla