ishepard

[ishepard]

We thank the reviewers for their comments! We updated the replication package to incorporate their feedback.

The new version can be found here: https://doi.org/10.5281/zenodo.3676517

We now answer each question individually:

blind-reviewer-2

I was using a Python virtual environment because I wanted to isolate this test, however, the program does not run if you follow your steps in a virtual environment. Perhaps you should investigate why that is. Replicating code within Python virtual environments is a popular method.

We re-run the experiment using virtualenv and we were able to run it. The commands we used to run it are:

   > virtualenv -p python3 venv
   ...
   Installing setuptools, pip, wheel...done.
   > source venv/bin/activate
   (venv) > pip install -r requirements.txt
   ...
   Successfully installed Flask-1.1.1 Jinja2-2.11.1 MarkupSafe-1.1.1 Werkzeug-1.0.0 click-7.0 itsdangerous-1.1.0 mistune-0.8.4
   (venv) > export FLASK_APP=run.py
   (venv) > flask run
    * Serving Flask app "run.py"
    * Environment: production
      WARNING: This is a development server. Do not use it in a production deployment.
      Use a production WSGI server instead.
    * Debug mode: off
    * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

It would be great if you could let us know whether it works for you. Thank you!

There is no explanation how to run the R script for people who have never used R before.

We modified the README.md file with the instructions on how to run the script from the terminal. We also recommend the reader to use RStudio to understand the analysis we did.

There is no requirements.R file, so the user has to manually install every dependency themselves. For R, this was particularly frustrating. Providing such a file would add a lot of value to this repository.

We modified the beginning of file regression.R so that it starts by checking the presence of the packages necessary to run our script. If they are not there, we automatically install all the missing packages.

gh-artifact-1763

The gist of the paper is participants and code review. However, the replication package did not give a table of participant characteristics i.e. to recreate Fig 2 in the paper, or to allow for replication by excluding certain characteristics like experience. Is there a reason this is omitted (or I missed it ..)?

We modified the file regression.R so that it outputs the plots of Figure 2 as well as saves them automatically in a 3-page pdf called distribution.pdf.

The second aspect is the code itself. I think it is available but buried in the tool's resources directory. To me these are a main replication artifact but currently they seem to be in undocumented text files, notwithstanding they are java code. Is it possible to somehow characterize these examples? I can see a big question being "how representative is the type of code you ask them to review". Currently that question is hard to answer.

In the replication package, we added a directory called code patches/, with inside the 2 Java files that were used in the experiment. For each file, we show the version without bugs as well as the version with the injected bugs and the reviewer's comment that is visible for the treatment groups.