recall-study

[minkull]

https://github.com/researchart/rose6icse/tree/master/submissions/available/recall-study https://github.com/researchart/rose6icse/tree/master/submissions/reusable/recall-study

Name	Email	Github ID
Li Sui	l.sui@massey.ac.nz	happylee
Jens Dietrich	jens.dietrich@vuw.ac.nz	jensdietrich
Amjed Tahir	a.tahir@massey.ac.nz	amjedtahir
George Fourtounis	gfour@di.uoa.gr	gfour

Corresponding Author: Li Sui (l.sui@massey.ac.nz)

Note to reviewers: these authors want multiple badges

• available
• reusable

[ICSE20ArtifactAnonymousReviewer]

The authors apply for the badges reusable and available.

*Reusable Functional: I could not run the artifact on my machine, by following the instructions in the install.md.

The following issues are solvable easily:

• For installation guide, you can add the step to navigate into the folder before calling docker build.
• Add the information that docker needs to run in Linux-container mode.

With these changes I could build the docker container successfully. However, 'docker run' did not work, even after substituting 'realpath' with the absolute path on my machine. I receive the following error message: "docker: Error response from daemon: invalid mode: /recall/data."

Note that my machine is running a Windows 10 with docker version 2.2.0.0. I hope the second reviewer will get the system to run.

Documentation: The readme file provides a good first overview over the artifact. I like the provided process overview.

Structure facilitating reuse and repurposing: The source code is commented, which is good.

License (ok): The license is EPL. It is indicated with the published artifact.

*Available Archival repository/DOI: The artifact is available on Bitbucket. However, there is no DOI or comparable identifier provided with the artifact. I propose to use an archiving service comparable to Zendo (https://zenodo.org/, https://guides.github.com/activities/citable-code/).

[li-sui]

Dear reviewer @ICSE20ArtifactAnonymousReviewer

To mount a local disk in Windows, please run docker run -i -v //c/Users/{user}/recall-study-artefact/data://recall/data -t recall

[timm]

@happylee: 👍 . but add that DOI please

@ICSE20ArtifactAnonymousReviewer : please advise if you can now assess this artifact

[ICSE20ArtifactAnonymousReviewer]

Dear @happylee, thank you for the new command. I now got a step further, until here:

root@...:/recall# dir addNocallsiteTag_script clean.xml java preAnalysis_script property.sh unreflectTests_script apache-ant-1.9.7 data jdk1.8.0_144 processCCTs_script runDoop_script xcorpus build.sh findFNs_script libs produceStats_script runTest_script

root@...:/recall# ./build.sh _test bash: ./build.sh: /bin/bash^M: bad interpreter: No such file or directory

Any over step I am missing, e.g. navigating into one of these folders?

[li-sui]

@ICSE20ArtifactAnonymousReviewer It must be an encoding problem. try bash build.sh _test

For other commands, you have to replace ./ with bash command. e.g. ./unreflectTests_script/unreflect-tests.sh _test -> bash unreflectTests_script/unreflect-tests.sh _test

[ICSE20ArtifactAnonymousReviewer]

@happylee: tried that and get now the following:

root@...:/recall# bash build.sh _test build.sh: line 2: $'\r': command not found build.sh: line 7: syntax error near unexpected token $'{\r'' 'uild.sh: line 7:import_property(){

[li-sui]

@ICSE20ArtifactAnonymousReviewer did you open build.sh with a Windows text editor? The script is written on a Linux system. If you have a sublime text or similar, change the Line Encoding to Unix style.

Or re-download the repository and re-build the artifact. Use cat build.sh to view the content.

[ICSE20ArtifactAnonymousReviewer]

@happylee: Not that I know. However, just to make sure, I just redid everything, starting from cloning the whole repository to a new directory and rebuilding the docker container, with the same result.

root@...:/recall# cat build.sh

!/bin/bash

import properties

${XCORPUS_DATA_DIR}=where the xcorpus is

${RECALL_ROOT_DIR} =artefact working directory

${programs}=list of programs

import_property(){ source "${RECALL_ROOT_DIR}"/property.sh }

build all programs, including: generate drivers, run/record CCTs, reduce and filter CCTs, extract reachable methods from CCTs, build SCG with doop, extract reachable methods from SCG, diff reachable method to find FNs

build_all(){ for program in "${programs[@]}" do build_single "$program" done } ... (just showing the first lines here).

[ICSE20ArtifactAnonymousReviewer]

Sorry that was auto-formated - the fat lines have a "#" in front of them...

[li-sui]

Dear @ICSE20ArtifactAnonymousReviewer

We suggest you to use a Linux platform, perhaps a VM. We have tested on Ubuntu 18.04 and it works fine. DOI is added to the repository: https://doi.org/10.5281/zenodo.3672128

[ICSE20ArtifactAnonymousReviewer]

In this case I propose we wait for the second reviewer and hope that he gets it to run.

[JonDoe-ArtifactsTrack]

i carried out the steps indicated on a Linux machine (debian testing), following to the @ICSE20ArtifactAnonymousReviewer comments and @happylee answers. The docker build was successful. Docker run was OK. Run the Artifact step: I did not run the artifacts as the estimated time was 45 days, while the process for a small set of programs requested too much memory that i did not have available. In each case i launched the run phase and stopped it at a certain point.

so for the badges: Available is Ok because it is functional and the artifacts are now available through a DOI added in the respository. Also, there is a detailed explanation for all the steps to carry out for th e installation and also for executing the programs. So i agree that it is reusable as well.

OK for badges requested on my side.

[minkull]

Both reviewers agree with available and @JonDoe-icse20Artifacts managed to run, recommending reusable. Awarding Available+Reusable.