hasabnis_eissec

[mcmillco]

Very Brief Summary

The artifact is an executable implementation of the EISSEC tool.

Insightful

The insightfulness of the idea is established in the research paper with which this artifact is associated. The artifact does not deviate from the idea.

Useful

The artifact makes it much easier to extract an RTL representation of code for use in symbolic execution. This task is currently quite complicated as it involves modifying the internals of GCC (or a manual process). This tool automates several key components of this task.

Usable

The artifact is implemented as a Docker component, which certainly makes the installation process easier than it might be otherwise. The paper also gives usage examples, include specific commands. However, the artifact does not include a virtual machine. The VM is arguably not necessary with the Docker component, but it is still advisable since future researchers may try to use the tool 5+ years from now, when dependencies are much harder to resolve.

It should also be noted that this paper is not quite in the right format. The paper needs to be condensed to one page for publication. The scorecard sheet is currently mixed in with the paper contents for a total of two pages.

[ghost]

This artifact describes an automated approach to extract semantics model from GCC’s code generator.

Insightful:

The idea of the paper is great and timely but I find the work not very well motivated. Also, the problem is not clearly stated and the gap in prior works in not well explained.

Useful:

The authors claim that existing works to extract instruction set semantics are manual. And that their work overcomes this challenge by performing the task in a fully automated way.

Could you please provide examples of existing works that are manual along with their references?

What about the effectiveness of the approach? It is important to provide us with an idea about that.

What about the execution time? Could you please specify that in the paper?

Also, has the approach been evaluated previously or tested on industrial or open-source projects? Do you have any published paper related to it?

Usable:

The paper is easy to understand.

The authors adhere to the FSE artifact track guidelines and explain the three required dimensions i.e. insightfulness, usefulness, and usability. But the explanation of these dimensions seems to be done in a very brief way, while they still have enough space that can be exploited.

A description of the artifact package has been provided along with the source code of the approach.

The authors also provided instructions on the installation of their artifact, which comes as a Docker image. Additionally, they put it available online.

Additional comments:

It is nice that the authors discussed the maturity of their approach even though I would prefer to see, prior to that, more details and discussions about the approach, its effectiveness and efficiency.

No single reference has been provided to any related work or to the current one.

Not sure why the authors structured the paper in this way. But the paper can be better restructured (if accepted).

While the idea of the paper is nice, I am more towards a reject. In effect, the paper is not well presented and motivated. Plus, I am not fully convinced how this artifact would be beneficial to the research community and—or practitioners. I am not an expert on that so may be other colleagues can perceive a value in that. I therefore stay open to change my score. For instance, I evaluate this submission as ‘maybe other’.

[nirhasabnis]

We would like to thank reviewers for their comments. We would definitely consider them while submitting a final version.

But we are not sure we understand the evaluation of 'maybe other'. Can we please get the scoring behind this evaluation? We could have definitely added motivation and system details, but then 2-page limit too strict for this purpose, and also those details are already covered in the main research paper. Instead, we preferred adding details about artifact, explanation of its output, which we believe are much more relevant for this paper.

We can definitely add missing motivation and details for the final version, but we are not sure we understand our artifact is evaluated 'maybe other' because lack of these details, or the binary image that we provided failed on the instructions that we supplied and the output that we explained. We would like to understand the scoring for this evaluation.

Thank you.

[mcmillco]

I concur with the authors here: there is only limited space for the "insightfulness" criterion in this paper, and we should generally assume that the FSE research track reviewers did their job evaluating that.

My vote is for a "gold" rating.

[timm]

@nirhasabnis We will mark this as "gold" and suggest that in future, for higher ratings in this format that the page1 material be more extensive.

[timm]

Note these labels are still "under discussion" and are still subject to change prior to the final notifications Friday.

[shaiduc]

[Recommendation]: Maybe Gold

The paper does not follow the instructions provided in the CFP. The first page must describe the artifact and the scorecard dimensions, and the second page must show how to use the artifact. In this paper, the first page includes instructions about how to use the artifact.

[Insightful]

• Timely: Some important tools rely on binary analysis, which can be hard to implement. This tool automatizes the process, making it an important resource for both developers and researches.
• Makes researchers smarter: It supports a difficult and time-consuming task (i.e. binary instrumentation), so it can make researchers more efficient.

[Useful]

• Serves a useful purpose?: Definitely. It supports an important task, which is in turn used by relevant software applications.
• Serves a purpose that would otherwise be tedious, prolonged, awkward, or impossible? Yes. Instruction-semantic extraction usually involves manual effort. An automatic process will decrease time and effort invested in this activity.
• Cost-effective? Yes. Developers do not need to spend a lot of time setting the tool up.

[Usable]

• Easy to understand? The output seems to be understandable only by experts. The usage of the tool is simple.
• Accompanied by tutorial notes? There is a README file available in the main web page of the tool.
• Easy to download: Yes. The entire system is provided as a tarball and as a Docker container.
• Available in a virtual machine image? Yes. It uses Docker.
• Supported by configuration management tools to permit easy updates? The virtual machine was created using Docker.

Other comments:

• The Introduction section shown in the following link http://seclab.cs.sunysb.edu/seclab/eissec/ is the same as the abstract of the following paper: “Lifting Assembly to Intermediate Representation: A Novel Approach Leveraging Compilers”, published at ASPLOS ’16; this seems to be a mistake.
• The link to the FSE research track paper listed here: http://seclab.cs.sunysb.edu/seclab/eissec/ is broken.
• The list of instructions described in the item "Using EISSEC Docker image" (Section 2) seems to be incomplete. After running the command "docker exec..." the following error was given: Error response from daemon: Container XXXXXX is not running However, after entering the following command: "docker start eissec" , the exec command worked without problem. It is then recommended to include the "docker start eissec" command in the list of instructions.

[timm]

FYI @nirhasabnis : we are moving to final decisions now. Regarding @shaiduc's comments on the format of the description page for this artifact: please take care to revise that artifact description in the final document sent to the FSE proceedings.