Closed minkull closed 4 years ago
Hi Authors,
Overall, you repository is well done. The replication package was easy to download and traverse. The code is well documented, which makes reading through it a pleasure. As someone who has never written in R, I was pleasantly surprised with the R language using your comments to walk me through what the code was doing (and why). Given the nature of your repository in having both a UI tool and analysis scripts, I am impressed by the ease-of-use. I was able to get the UI loaded and the analysis scripts to run. There were some difficulties encountered, and those are listed below in the detailed comments. Future parties interested in using your repository would likely appreciate you addressing those difficulties. With these comments in mind, I find it clear that this repository is at least "Reusable."
The authors have uploaded the data to Zenodo. This resource means that the data will be available for an extended period of time for all interested parties to use. With this included, I find it clear that this repository is at least "Available".
Installation Comments
results.csv
file is generated, we can run the R script regression.R
." is not sufficient.Documentation
Dataset Availability: Uploaded to Zenodo License: MIT License
This is a nicely documented replication package. I had no trouble running the scripts or the sample tool.
That being said I think there are two omissions with respect to the paper's claims and replicating the paper.
The gist of the paper is participants and code review. However, the replication package did not give a table of participant characteristics i.e. to recreate Fig 2 in the paper, or to allow for replication by excluding certain characteristics like experience. Is there a reason this is omitted (or I missed it ..)?
The second aspect is the code itself. I think it is available but buried in the tool's resources
directory. To me these are a main replication artifact but currently they seem to be in undocumented text files, notwithstanding they are java code. Is it possible to somehow characterize these examples? I can see a big question being "how representative is the type of code you ask them to review". Currently that question is hard to answer.
@ishepard @gulcalikli @sback
you have one reviewer already willing to say reusable
but another reviewer who wants some more details
can u supply those additional details? in a. pull request? next few days?
currently we are at "no badge" and will remain there till authors offer more details
Hi @timm! We are working on the feedback. We will release the new version of the package today! Thanks!
Just created the PR. See #192.
Thank you!
@blind-reviewer-2 recommends reusable. @gh-artifact-1763, please could you check if the revisions are satisfactory?
As a side note, the replication package is available on Zenodo, hence we are seeking for Available and Reusable.
What I first understood from the doc is that Reusable implies Available, but from the issues already closed I see that there are packages that only have Reusable and not Available.
Since our package is actually available with a DOI, I think it also deserves the Available badge. Please let me know if you disagree with me.
Thank you!
at this stage, I am almost ready to endorse "available" and "reusable" (unless @gh-artifact-1763 wants to veto "available..." )
But first, to get available, @ishepard, please point to where in the artifact files in rose6icse there is the DOI
I see this in their README: The replication package can be downloaded at this URL: https://doi.org/10.5281/zenodo.3676517
and endorse this artefact as "Available".
https://github.com/researchart/rose6icse/tree/master/submissions/reusable/ishepard
seeking "reusable"