----------------------- REVIEW 1 ---------------------
PAPER: 41
TITLE: Navigating Program Flow in the IDE
AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
----------- Review -----------
This paper summarizes a new tool that inserts additional navigable links into Eclipse, so that programmers can more easily navigate from variables and methods back to the code that references those variables and methods.
As far as I can tell, this tool has the same function as "Find References" in Eclipse. This existing feature displays a list of referring code in a search output panel, and it even highlights occurrences in the code. Visual Studio also has a similar feature. If the proposed tool differs from Find References in some fashion, it would be helpful have a much clearer explanation in the Introduction and then a detailed walkthrough in Section IV. (#110)
It is good that the paper includes some empirical support for the utility of this tool, but its inability to meet the effectiveness of the existing tool for task 2 (Section VI / Branching and Backtracking) suggests that it needs further work. (#118)
In terms of presentation, I'd suggest replacing Table I with either a figure or a table that shows results or impact, rather than information about participant demographics. Most readers will be more interested in the tools' effectiveness than in the participants' background. (#120)
(After the author response)... Thank you for acknowledging our feedback and especially for explaining the differences relative to Find References. I'm increasing my rating slightly, but I still don't see enough novelty in this work to justify arguing on behalf of its acceptance.
----------------------- REVIEW 2 ---------------------
PAPER: 41
TITLE: Navigating Program Flow in the IDE
AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
----------- Review -----------
I thank the authors for their rebuttal text. I am not convinced this paper does justice to the idea/the work sufficiently well to accept.
Pros:
-nice human-centric paper
-nicely written
-idea is interesting
-evaluation preliminary but promising
Cons:
-hard to determine just what Flower does (#119)
-more detail on Flower ; less on motivation & related work might help (#119)
-evaluation is preliminary / nuclear just what learned (#118)
The problem motivation is real - I’ve had the same experience many times when trying
to navigate in Eclipse (and many other IDEs). Thus I can see the potential benefits
of this approach.
However I found it hard to judge just what the tool offers / how it works / its
strengths and limitations. In part this is due to very limited explanation/illustration
of the tool. I would suggest more emphasis on this and abet less on the scenario and
related work would help. Perhaps reduce even the evaluation explanation slightly too. (#119)
The evaluation as admitted by the authors is very preliminary. I wondered given the
limited idea of strengths and weaknesses from this just how much the paper really
shows about the proposed approach (good or bad) - perhaps again more focus on the
“interesting idea” via more on the tool/usage example would be better vs evaluation
detail? (#118)
This would make a nice demo paper…
----------------------- REVIEW 3 ---------------------
PAPER: 41
TITLE: Navigating Program Flow in the IDE
AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
----------- Review -----------
This paper introduces a simple new way to navigate some aspects of
control flow, through inserting clickable names in Eclipse. This
helped a little with a few tasks.
I was disappointed that a more interesting navigation technique was
not tried that might help with more complex navigations - maybe in a
future paper. Since this is a short paper, however, I feel the
presented results are sufficent for acceptance. (#117)
I also didn't understand how the presented techniques required
"powerful program analysis techniques" when it only needed to find out
the direct callers and callees - isn't that available from a
rudimentary program analysis? (#116)
How would users go back after clicking on one of the links if they
wanted to look at the next one? You said: "P6 attempted to use
Eclipse's built-in back buttons to backtrack, but still had difficulty
reorienting himself." -- why? (#108)
The need for frequent returning to where one was and use of regular
scrolling to navigate, and the general need for better navigation, was
well reported in this previous study:
Andrew J. Ko, Brad A. Myers, Michael Coblenz, and Htet Htet Aung. "An
Exploratory Study of How Developers Seek, Relate, and Collect Relevant
Information during Software Maintenance Tasks", IEEE Transactions on
Software Engineering. Vol. 32, No. 12, Dec, 2006. pp. 971-987. (#115)
The user test reported here is rudimentary and doesn't include any
statistics. Why didn't you do a more systematic test? (#114)
The idea of coloring the links based on ones that were visited is what
the WWW used to do, which everyone recognized was great, and now is
sadly mostly gone.
Writing issues:
Maybe using the name "Hillary" as your example isn't a good choice
this year! Especially since the conference will be just before the US
election. (#113)
The comment: "Though Hillary is fictional, her story is based on the
experiences of real developers we observed in a previous study [6]" --
should be moved to the TOP of the example. (#113)
typo: "Call Hirerchy" (#112)
The acknowledgement you have a footnote won't be adequate for NSF -
you need to devote the appropriate space for the acknowledgements. (#111)
----------------------- REVIEW 1 --------------------- PAPER: 41 TITLE: Navigating Program Flow in the IDE AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
OVERALL EVALUATION: -1 (weak reject) REVIEWER'S CONFIDENCE: 4 (high)
----------- Review ----------- This paper summarizes a new tool that inserts additional navigable links into Eclipse, so that programmers can more easily navigate from variables and methods back to the code that references those variables and methods.
As far as I can tell, this tool has the same function as "Find References" in Eclipse. This existing feature displays a list of referring code in a search output panel, and it even highlights occurrences in the code. Visual Studio also has a similar feature. If the proposed tool differs from Find References in some fashion, it would be helpful have a much clearer explanation in the Introduction and then a detailed walkthrough in Section IV. (#110)
It is good that the paper includes some empirical support for the utility of this tool, but its inability to meet the effectiveness of the existing tool for task 2 (Section VI / Branching and Backtracking) suggests that it needs further work. (#118)
In terms of presentation, I'd suggest replacing Table I with either a figure or a table that shows results or impact, rather than information about participant demographics. Most readers will be more interested in the tools' effectiveness than in the participants' background. (#120)
(After the author response)... Thank you for acknowledging our feedback and especially for explaining the differences relative to Find References. I'm increasing my rating slightly, but I still don't see enough novelty in this work to justify arguing on behalf of its acceptance.
----------------------- REVIEW 2 --------------------- PAPER: 41 TITLE: Navigating Program Flow in the IDE AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
OVERALL EVALUATION: -1 (weak reject) REVIEWER'S CONFIDENCE: 5 (expert)
----------- Review ----------- I thank the authors for their rebuttal text. I am not convinced this paper does justice to the idea/the work sufficiently well to accept.
Pros: -nice human-centric paper -nicely written -idea is interesting -evaluation preliminary but promising
Cons: -hard to determine just what Flower does (#119) -more detail on Flower ; less on motivation & related work might help (#119) -evaluation is preliminary / nuclear just what learned (#118)
The problem motivation is real - I’ve had the same experience many times when trying to navigate in Eclipse (and many other IDEs). Thus I can see the potential benefits of this approach.
However I found it hard to judge just what the tool offers / how it works / its strengths and limitations. In part this is due to very limited explanation/illustration of the tool. I would suggest more emphasis on this and abet less on the scenario and related work would help. Perhaps reduce even the evaluation explanation slightly too. (#119)
The evaluation as admitted by the authors is very preliminary. I wondered given the limited idea of strengths and weaknesses from this just how much the paper really shows about the proposed approach (good or bad) - perhaps again more focus on the “interesting idea” via more on the tool/usage example would be better vs evaluation detail? (#118)
This would make a nice demo paper…
----------------------- REVIEW 3 --------------------- PAPER: 41 TITLE: Navigating Program Flow in the IDE AUTHORS: Chris Brown, Justin Smith, Tyler Albert and Emerson Murphy-Hill
OVERALL EVALUATION: 1 (weak accept) REVIEWER'S CONFIDENCE: 5 (expert)
----------- Review ----------- This paper introduces a simple new way to navigate some aspects of control flow, through inserting clickable names in Eclipse. This helped a little with a few tasks.
I was disappointed that a more interesting navigation technique was not tried that might help with more complex navigations - maybe in a future paper. Since this is a short paper, however, I feel the presented results are sufficent for acceptance. (#117)
I also didn't understand how the presented techniques required "powerful program analysis techniques" when it only needed to find out the direct callers and callees - isn't that available from a rudimentary program analysis? (#116)
How would users go back after clicking on one of the links if they wanted to look at the next one? You said: "P6 attempted to use Eclipse's built-in back buttons to backtrack, but still had difficulty reorienting himself." -- why? (#108)
The need for frequent returning to where one was and use of regular scrolling to navigate, and the general need for better navigation, was well reported in this previous study:
Andrew J. Ko, Brad A. Myers, Michael Coblenz, and Htet Htet Aung. "An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks", IEEE Transactions on Software Engineering. Vol. 32, No. 12, Dec, 2006. pp. 971-987. (#115)
The user test reported here is rudimentary and doesn't include any statistics. Why didn't you do a more systematic test? (#114)
The idea of coloring the links based on ones that were visited is what the WWW used to do, which everyone recognized was great, and now is sadly mostly gone.
Writing issues:
Maybe using the name "Hillary" as your example isn't a good choice this year! Especially since the conference will be just before the US election. (#113)
The comment: "Though Hillary is fictional, her story is based on the experiences of real developers we observed in a previous study [6]" -- should be moved to the TOP of the example. (#113)
typo: "Call Hirerchy" (#112)
The acknowledgement you have a footnote won't be adequate for NSF - you need to devote the appropriate space for the acknowledgements. (#111)