ishepard

[minkull]

https://github.com/researchart/rose6icse/tree/master/submissions/reusable/ishepard

1. Davide Spadini (corresponding author):
   • email: spadini.davide@gmail.com
   • GitHub ID: ishepard
2. Gul Calikli:
   • email: calikli@chalmers.se
   • GitHub ID: gulcalikli
3. Alberto Bacchelli:
   • email: bacchelli@ifi.uzh.ch
   • GitHub ID: sback

seeking "reusable"

[blind-reviewer-2]

Hi Authors,

Overall, you repository is well done. The replication package was easy to download and traverse. The code is well documented, which makes reading through it a pleasure. As someone who has never written in R, I was pleasantly surprised with the R language using your comments to walk me through what the code was doing (and why). Given the nature of your repository in having both a UI tool and analysis scripts, I am impressed by the ease-of-use. I was able to get the UI loaded and the analysis scripts to run. There were some difficulties encountered, and those are listed below in the detailed comments. Future parties interested in using your repository would likely appreciate you addressing those difficulties. With these comments in mind, I find it clear that this repository is at least "Reusable."

The authors have uploaded the data to Zenodo. This resource means that the data will be available for an extended period of time for all interested parties to use. With this included, I find it clear that this repository is at least "Available".

Given the current state of the repository, I recommend a badge of "Reusable"

Detailed Comments

Reusable Badge (+ Functional)

Installation Comments

• I was using a Python virtual environment because I wanted to isolate this test, however, the program does not run if you follow your steps in a virtual environment. Perhaps you should investigate why that is. Replicating code within Python virtual environments is a popular method.
• I got the online tool running. Well done. Steps were straight forward.
• There is no explanation how to run the R script for people who have never used R before.
  • "Once the results.csv file is generated, we can run the R script regression.R." is not sufficient.
• There is no requirements.R file, so the user has to manually install every dependency themselves. For R, this was particularly frustrating. Providing such a file would add a lot of value to this repository.
  • If you do not wish to provide such a file, then please provide a list of the required dependencies to install before listing the command to run the R script.

Documentation

• Nice touch with the already-generated files to compare against

Available Badge

Dataset Availability: Uploaded to Zenodo License: MIT License

[gh-artifact-1763]

This is a nicely documented replication package. I had no trouble running the scripts or the sample tool.

That being said I think there are two omissions with respect to the paper's claims and replicating the paper.

The gist of the paper is participants and code review. However, the replication package did not give a table of participant characteristics i.e. to recreate Fig 2 in the paper, or to allow for replication by excluding certain characteristics like experience. Is there a reason this is omitted (or I missed it ..)?

The second aspect is the code itself. I think it is available but buried in the tool's resources directory. To me these are a main replication artifact but currently they seem to be in undocumented text files, notwithstanding they are java code. Is it possible to somehow characterize these examples? I can see a big question being "how representative is the type of code you ask them to review". Currently that question is hard to answer.

[timm]

@ishepard @gulcalikli @sback

you have one reviewer already willing to say reusable

but another reviewer who wants some more details

can u supply those additional details? in a. pull request? next few days?

[timm]

currently we are at "no badge" and will remain there till authors offer more details

[ishepard]

Hi @timm! We are working on the feedback. We will release the new version of the package today! Thanks!

[ishepard]

Just created the PR. See #192.

Thank you!

[minkull]

@blind-reviewer-2 recommends reusable. @gh-artifact-1763, please could you check if the revisions are satisfactory?

[ishepard]

As a side note, the replication package is available on Zenodo, hence we are seeking for Available and Reusable.

What I first understood from the doc is that Reusable implies Available, but from the issues already closed I see that there are packages that only have Reusable and not Available.

Since our package is actually available with a DOI, I think it also deserves the Available badge. Please let me know if you disagree with me.

Thank you!

[timm]

at this stage, I am almost ready to endorse "available" and "reusable" (unless @gh-artifact-1763 wants to veto "available..." )

But first, to get available, @ishepard, please point to where in the artifact files in rose6icse there is the DOI

[blind-reviewer-2]

I see this in their README: The replication package can be downloaded at this URL: https://doi.org/10.5281/zenodo.3676517 and endorse this artefact as "Available".