Submission 74

[Edkamb]

This issue will be used for the communication concerning the testing phase of submission 74.

[ReviewerFase2022]

Documents

The submission contains four documents:

1. The abstract (available on EasyChair), which also contains the submitted paper.
2. A file named abstract.pdf inside the artifact zip file on Zenodo.
3. A file named README.pdf inside the artifact zip file on Zenodo, which seems to be a user guide.
4. A file named README.md inside the artifact zip file on Zenodo, which seems to be the markdown version of the file README.pdf.

The first part of document 1 (the abstract on EasyChair) and document 2 contain similar information, but they are not exactly the same. I used document 1 for the test phase since it is on EasyChair. Between documents 3 and 4, I used document 3. If the authors disagree with this decision, I kindly ask them to let me know.

According to document 1, Section 1.1 (STRUCTURE OF THE DIRECTORY CONTAINING THE ARTIFACT), there are four directories under ResFuz/evaluation: BUILD, COCCI, FUZZ, and OUT. The first three directories exist on this path. But, I found a directory named PLOT, instead of OUT. Is that a typo?

Reading the submitted abstract on EasyChair and README.pdf, it seems all claims and results of the paper can be replicated with the artifact. I kindly ask the authors to let me know if it is not true because I was not able to check it myself due to the following reasons.

Test Phase Failed

I was not able to run the test phase because of the following problems.

According to the abstract, the first step of the test phase is to run the shell script build_jasper.sh existing in the directory ResFuz/evaluation/BUILD. This file does not exist.

The second step of the test phase is to run the shell script run_ResFuz_jasper.sh existing in the directory ResFuz/evaluation/FUZZ. This file does not exist either.

The third step of the test phase says:

“If you want to compare its performance with MemLock or AFL, you can open another terminal and run the script ./run_MemLock_jasper.sh or ./run_AFL_jasper.sh.”

These two files do not exist in the directory ResFuz/evaluation/FUZZ.

The abstract on EasyChair says:

“Please read README_ResFuz_v2.md for more details under path-to-ResFuz/README_ResFuz_v2.md”.

I did not find the file README_ResFuz_v2.md under path-to-ResFuz/.

Other problems

According to README.pdf, one must run the shell script ./INSTALL.sh to build ResFuz. Running this shell script results in an error complaining the commands autoreconf and git do not exist (I think because autoconf and git are not installed on FASE 2022 virtual machine).

Furthermore, this shell script (./INSTALL.sh) downloads several Ubuntu and Python Packages from the internet. Also, running ResFuz/tool/install_llvm.sh starts downloading LLVM from the internet. According to the FASE Artifact Evaluation 2022 instructions:

“The artifact evaluation committee will be instructed not to download software or data from external sources. Any additional software required by your artifact must be included in the .zip file and the artifact must provide instructions for the installation.”

What I get from the instructions is that downloading packages from the internet is not allowed. Thus, the authors must include these packages in the artifact zip file and modify their shell scripts accordingly.

In addition, running the shell script ResFuz/tool/install_llvm.sh results in the following error because FASE 2022 virtual machine does not have git installed.

+ git clone https://github.com/ICSE2020-MemLock/LLVMlib_asan.git /home/fase2022/Desktop/Files/FASE2022_ResFuz/ResFuz/tool/MemLock/LLVMlib_asan_6.0.1
./install_llvm.sh: line 26: git: command not found

Please fix these problems and make sure the test phase runs on FASE 2022 virtual machine with no errors.

[phd-emmett-brown]

I used the first part of the document available on EasyChair which also contains the submitted paper. To run the artifact I've used the provided virtual machine.

Some issues regarding the instructions to launch the experiments which were not included in the instructions document:

• I needed to install git.

• There was no build_jasper.sh file to test the artifact and launch the corresponding experiment. Instead, I have launched

build_jasper_buffer.sh build_jasper_heap.sh

At the first attempt, there was some incompatibility when launching the experiments, and I had to enable the modification permissions of the core-pattern file, following the instructions provided in this link:

How to permanently edit the core pattern file

After that, I executed the corresponding command (again, this is not exactly what appears in the instructions document):

run_ResFuz_jasper_buffer.sh

The experiment is now running and everything seems to be working now.

Please, include these changes in the revised artifact and those found by the other reviewers before starting the next phase.

Thanks for your work!

[lqchen]

We thank the AE reviewers for their useful feedback. We have revised the artifact according to the feedback in the following aspects.

1. We have revised the abstract and have updated it in the Easychair (and delete the abstract.pdf in the .zip file on Zenodo). Please use the new URL mentioned in the revised abstract to visit our new artifact.

2. Indeed, the INSTALL.sh script in the previous artifact needs to download extra packages from Internet. We have now included all additional software required by our artifact in the new .zip file and have modified the installation scripts.

3. We have revised the README.md (and README.pdf) in the artifact, removing typos and inconsistencies (e.g., the PLOT or OUT problem, the wrong file names in the previous README such as build_jasper.sh which should be build_jasper_heap.sh or build_jasper_buffer.sh, and similarly run_ResFuz_jasper.sh which should be run_ResFuz_jasper_heap.sh or run_ResFuz_jasper_buffer.sh). Moreover, we have made it more clearly the instructions to lunch and stop fuzzing in the README, such as how to edit the core pattern file.