Sui_SUPA

[mvdbrand]

The artefact description describes clearly the goals of the artefact. It provides a clear description on how to reproduce the results in the accepted paper. The artefact geared towards the paper, although the authors claim that their tool has a growing user base, both from academia and industry. It would be nice if the authors could give, for instance, some statistics on downloads of their tool, to strengthen this claim. The authors provided a SUPA web page, containing similar information as the artefact description. The artefact description contains a detailed description of the required platform and the project layout. The SUPA web page does not provide more information on more experiments than presented in the description. I was unfortunately not able to do the installation myself because of incompatible hardware.

I would classify this artefact as: ? maybe platinum

[mcmillco]

Very Brief Summary

The artifact is a reproducibility package for the associated FSE paper.

Insightful

The insightfulness of the underlying approach is established in the FSE paper. This artifact is timely because it helps other researchers to use the state-of-the-art approach that the paper describes.

Useful

The implementation is built into an existing framework, which increases its usefulness since there are existing users of that framework who may benefit.

Usable

The artifact meets all the criteria for usability: the implementation is built into an existing framework, a website clearly describes how to set up and use the artifact (including the commands to execute), and a VM is provided with everything preinstalled.

This artifact should be accepted. The only caveat is that the submitted PDF is not in the correct format: the proceedings only allow one page. The Artifact Package section is on the same page as the scorecard. I think the authors should have a chance to fix it, since the artifact looks strong.

[ghost]

This artifact includes an implementation of a tool called Strong UPdate Analysis (SUPA) for the computation of points-to information on-demand via value flow refinement.

Pros:

The paper is interesting, timely, and relevant for the FSE 2016 artifact track.

The artifact corresponds to a well motivated work that has been already published in FSE 2016. I therefore believe that this artifact would be very valuable for replications and comparison purposes.

The tool achieves a precision similar to a whole-program analysis by consuming only 0.19 seconds, which makes it cost-effective.

The paper adheres to the FSE 2016 artifact guidelines. It clearly discusses the three required dimensions: insightfulness, usefulness, and usability. Plus, it is easy to understand.

The authors provided an artifact package along with its detailed description. The later includes a checklist of their package, guidelines to reproduce the results, and Wiki documents to understand how its works and—or develop further analyses.

The authors have also provided a virtual machine image, which contains the SUPA releases, together with scripts and benchmarks.

They provided links for additional information about SUPA including its websites (http://www.cse.unsw.edu.au/˜corg/supa) and the Wiki site of the SVF framework on which it is based (http://unsw-corg. github.io/SVF).

Cons:

I was expecting to see a brief discussion about the challenges and—or limitations associated with this artifact. May be for space reasons, the authors could not mention more details but it would be great if the authors highlight that in the paper.

A reference is required for the authors’ FSE 2016 paper where the related approach is described. Please add it in the paper. You may also want to add a link to the paper or its Web page (if any).

More references are needed in the paper especially in the first sections (where the authors introduce the context, motivation, and insightfulness). The authors can summarize details about the artifact package to gain some space for more discussion/references. And put the additional details about the artifact package is an appendix for example (if needed).

Overall, I believe this work is well motivated and strong and its corresponding artifact can make a nice contribution to the FSE 2016 artifact track. This submission could be a candidate for a Platinum award but let’s see if other pc members share the same opinion :)

[mcmillco]

I concur with a "platinum" rating for this artifact.

[mvdbrand]

Me too.

[timm]

Note these labels are still "under discussion" and are still subject to change prior to the final notifications Friday.