jssmith1
commented
7 years ago R1 reasons to reject:
- [x] The navigation of the methods is not as well supported yet as the paper claims
- [x] tool breaks in some circumstances
- [x] tool's minimalism fails in complex systems
R2
- [x] tool only useful for very simple tasks
- [x] we are sexist
R3
- [x] remove "fired for oversight"
- [x] clarify why motivating example fails and how our approach fixes them
- [x] Program analysis -> accuracy?
- [x] p1 the the
----------------------- REVIEW 1 --------------------- PAPER: 10 TITLE: Flower: Navigating Program Flow in the IDE AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
OVERALL EVALUATION: -1 (weak reject)
----------- Review ----------- This paper introduces Flower, a plugin Eclipse for navigating program (data and control) flow utilizing a minimalistic UI.
Much of the strength of this paper lies in its low barrier to adoption. The tool appears to be specifically engineered to appeal to "real life" scenarios in which cumbersome tools often do more harm than good, and simplicity is a key factor in determining if a developer utilizes a tool or not. The UI elements are intuitive, using hyperlinks. Moreover, the fact that there are some preliminary results from a user study on the tool greatly strengthens the submission. The use of "Roxanne"'s example is poignant and succinctly illustrates both the novelty of and demand for a tool like Flower.
The evaluation process described in the paper, even though it involves a small number of developers, is a great contribution that can help to further develop this tool and related tools. In this sense, it would be interesting to know the number of participants that declared to be familiar with the Eclipse IDE, and their opinion about the capabilities of Flower compared with the existing IDE tools.
Flower makes a great contribution to the data flow comprehension tasks. It is able to successfully highlight the provenance of the parameters of a method. By integrating this functionality into the main IDE editor, the developer might save a lot of time and cognitive effort when she tries to establish the execution flow of a piece of code.
However, the navigation of the methods is not as well supported yet as the paper claims. Using the provided virtual machine, the reviewer created a new class as follows:
public class Second{ public void foo(String filename){ Test app = new Test(); //The modifier of Test’s constructor was changed app.getQueries(filename); } }
Despite Flower being able to detect that the string parameter of the method Test.getQueries can be provided by the method Second.flow, the tool unfortunately was not able to detect that the method foo was calling the method getQueries. This test shows that the tool does not support full program navigation yet, but it definitely offers a new promising direction for the design of these tools.
Additionally, the plugin needs additional development to be properly integrated in the Eclipse IDE. Right now, the tool breaks the “Call Hierarchy” functionality, and after some code modifications the tool evidenced unexpected behavior. In fact, I had to restart the IDE several times to be able to finish the tests.
Another weakness of the tool is that the minimalistic approach only seems intuitive in some cases. In a very large system with significantly interconnected classes (indicative of a system that one would think necessitates the most tool support for evolution/maintenance), the minimalistic approach may not meet the needs of developers. Specifically, as shown in the user study, participants wanted the ability to navigate "forward" and "backward" to investigate various code branches. This weakness was largely addressed by the authors' note about future research in facilitating transition from the minimalistic tool to a more heavy-duty tool for complex or malicious cases. Even though the results were not statistically significant, qualitative information from the preliminary study seemed to indicate that this was due to experimental design limitations and not the tool itself. It would be very interesting to see a follow-up study on a larger number of SE tasks with varying degrees! of complexity in order to better understand the point at which minimalism becomes a detriment to navigation. It would also benefit the paper if there were more details on how the navigation works in some more complex situations. For example, if there are many child classes that override a method, what happens when one navigates to an invocation on the parent's method? Does the compile-time program analysis attempt to understand which child class is more likely using some sort of casting heuristic? Or is the user presented with a list of possibilities? Having a list of canonically problematic scenarios and how these scenarios are dealt with inside of the tool would be very interesting, and potentially helpful for future related research.
----------------------- REVIEW 2 --------------------- PAPER: 10 TITLE: Flower: Navigating Program Flow in the IDE AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
OVERALL EVALUATION: -1 (weak reject)
----------- Review ----------- Summary: This live tool demo paper presents Flow-er, a tool to improve program navigation in the IDE. The interface of the tool is relatively minimal and non-invasive, allowing better navigation of control-flow and data-flow for simple Java programs.
Tool/Materials evaluation: The authors included a link to a virtual machine, which is working, and a screencast for the video.
Evaluation: I did some work on navigation on program comprehension and I believe that the research the authors are doing is fundamental to improve the evident issues that concern navigation on modern IDEs. The problem is well known from years and I appreciate the effort done by the authors. The aim for a minimalistic interface, and in general all the design principles of Section II. However, I really feel that at the current status the kind of navigation tasks that are improved and where the tool is really effective seem to be limited to relationships between methods in the same class, which constitute a class of relatively easy navigation tasks. The example in the video screencast is of the same class and looks relatively simple, and failed to convince me that the tool really changes something significant in the developers' experience. The tasks on the evaluation are described in a too general way and I'm not sure how difficult they actually are. They actually seem simply some input sanitization/validation tasks (for which in many cases static analysis tools can solve the problem), and they are not really general navigation tasks. However, I appreciate the research effort and the particular topic, so I think it could make an interesting, stimulating live tool presentation.
As an orthogonal remark, that I did not use to decide for the fate of this submission, let me point out that there is something in your introduction that can be easily mistaken and felt as sexist. I use "mistaken" because I am sure it has not been done intentionally, yet I suggest the author to avoid it. Using a female character that is "fictional", while "her story is based on the experiences of real developers we observed in a previous study" to illustrate a behavior that is implicitly considered so unprofessional to cost her firing, in a domain like the one of software engineering which has huge gender problems, looks ugly in a scientific paper. I suggest to avoid that. Make it gender neutral, or do not cast to a specific, fictional character.
----------------------- REVIEW 3 --------------------- PAPER: 10 TITLE: Flower: Navigating Program Flow in the IDE AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
OVERALL EVALUATION: 1 (weak accept)
----------- Review ----------- The paper presents Flower, a tool for navigating data and control flow of a program in the Eclipse IDE. The navigation is happening while developers inspect the code without switching views. For example, when a developer looks at a method declaration and selects a formal parameter of a method, to tool provides links to methods that call this method and methods that are called with the formal parameter. The developer can then navigate and investigate the program flow. The tool is evaluated with a preliminary study in which eight graduate students used Flower to solve two tasks. Although results do not show statistically significant improvement in completion time and correctness when participants use the tool, the authors discuss the positive and negative results of the evaluation and propose ideas to further improve the tool. A demo of the tool and the material for the study are available online.
Suggested improvements:
Minor comments: