silva_whyWeRefactorDataset

[ctreude]

This is a very insightful, useful and usable dataset.

In addition to the corresponding paper being a really nice contribution to the refactoring research community, the authors have done a great job putting together this dataset: The website is clearly structured, the visual presentation is appealing, data is available in a navigable format and as JSON, and all the relevant data appears to be there. I particularly appreciate the amount of detail regarding the thematic analysis which shows themes before and after consensus. The level of detail provided here works great towards increasing a user's confidence in the process of determining the themes.

As the authors point out, similar work is pretty limited and the amount of detail available in this dataset along with the ease of use make me optimistic that this dataset will be used by other researchers.

I'm happy to recommend the inclusion of this dataset in the online repository, the social media campaign and the award session.

[hongyujohn]

The authors have done a good job in collecting and analyzing the refactor dataset. The dataset is detailed and well-structured. The dataset not only reveals the type of refactoring in programming practices but also includes motivations behind each refactoring. To conduct thematic analysis the authors collected feedback from programmers. The dataset is easy to use with clear web items and JSON files. In summary, collecting such a dataset is a good job, which could help researchers and practitioners better understand the refactoring activity. I hope the authors can continue maintaining it in the future to make it a better resource for refactoring research.

I recommend acceptance.

[yasutaka-kamei]

This paper provides the dataset that investigates the motivations behind specific refactoring operations conducted by developers. The dataset includes (1) a list of 539 commits with 1,411 refactoring identified by the RefactoringMiner and confirmed with manual inspection and (2) 222 commits with 463 refactoring and their motivations.

According to the CFP, artifact papers are expected to include:

• Insightful
  • Timely (i.e., addresses a problem that is most current and most pressing)? Yes. The dataset includes developer surveys and interviews to help researchers to investigate the motivations behind refactoring. The topic the dataset covers is very timely.
  • Makes researchers “smarter” in some way (e.g., identifies and fills some significant gap in prior work)? Yes. The dataset makes researchers smarter because they can better understand why developers examine refactoring by analyzing the dataset.
• Useful
  • Serves a useful purpose? Yes. The dataset can be used to investigate the motivations behind refactoring operations performed by developers.
  • Cost-effective? Yes. Future studies can save research effort for conducting manual inspection and developer survey by reusing this dataset. So the dataset is very cost-effective.
• Usable
  • Easy to understand? Yes. The dataset is organized very well and easy to understand.
  • Accompanied by tutorial notes? Yes. The 2nd page of the paper clearly explains what the dataset includes and how future studies can use the dataset.
  • Artifacts need not be executable but if so, are they: No. Artifacts are data sets. So, they are not executable.
  • Easy to download, install, or execute? Yes. The data set is available at GitHub. It's easy to download it. In addition, dataset in JSON format is also available and is easily used by other tools.
  • Available in a virtual machine image? No.
  • Available online? Yes. The data set is available at GitHub.
  • Supported by configuration management tools to permit easy updates? No. Future studies may want to update this dataset because they obtain more commits with their motivation by conducting additional developer survey. Therefore, if the web page (http://aserg-ufmg.github.io/why-we-refactor) includes information about "how to contribute to improve the dataset" (e.g., how to make a pull request), it could be helpful.
• Score I think that this paper is the best paper among the papers I reviewed in FSE2016 artifacts track. So I would like to say "Diamond" for this paper.

[timm]

Note these labels are still "under discussion" and are still subject to change prior to the final notifications Friday.

[danilofes]

First, we would like to thank all reviews for their comments. Second, we will address the following suggestion by Yasutaka Kamei:

Future studies may want to update this dataset because they obtain more commits with their motivation by conducting additional developer survey. Therefore, if the web page (http://aserg-ufmg.github.io/why-we-refactor) includes information about "how to contribute to improve the dataset" (e.g., how to make a pull request), it could be helpful.

This is a very interesting suggestion. We will include instructions in the website and in the camera-ready version of paper on how to contribute to improve the dataset via GitHub pull requests.