Reviewer comments

[Suvodeep90]

Reviewer 1:

• My major concern is about the experimental design and the definition of the defect models in this paper. (Update according to JIT and Released based setting)
• Specifically, it's not clear to me if this paper focused on release-based defect prediction or JIT defect prediction. Please clearly state at the beginning of the paper. (Update according to JIT and Released based setting)
• readers need strong justification and convincing statements that the proposed approach is sound and correct. Currently, the use of product metrics in this paper does not sound right to me. The prediction granularity of JIT models is at the commit level, and it's almost impossible to map the product metrics to each commit. That's why the original Kamei paper does not have product metrics. (modify section 3.1 to be more clear about data set creation)
• Step 3 in 3.1 is very hard to understand, thus it needs an overview diagram to convince reviewers and support future replication. my quick understanding is that the authors aim to generate product metrics for each file in each commit. So my question is that how can the authors aggregate these product metrics of multiple files to the commit level. (modify section 3.1 to be more clear about data set creation)
• the experimental setup of this paper is very different from the Rahman paper. (1) The process metrics in Table 2 are proposed by Kamei, but are not the same set that was used in Rahman et al. (2) Rahman et al focused on release-based DP, not JIT DP. (Update according to JIT and Released based setting)
• if the paper focused on JIT models, why not formulate the paper to replicate Kamei et al, instead of Rahman et al.
• The authors stated that "Only one paper argues that product metrics are best. Arisholm et al. [6] found one project where product metrics perform better". I don't think it is correct to draw conclusions from "one project" of the Arisholm paper. (Yes we agree, that's why this study)
• In 3.1 Collaboration, the authors used #pull requests as a proxy of collaboration. I don't think this is correct. Do the authors have any reference to support this? (add citations)
• Please report the ratios of bug-introducing commits as well in Table 4. (add a table including dataset details)
• Please include a reference to support the definition of "thick-slice" validation. (We don't have any citations. This is a new validation approach. Add details why it is useful kin JIT)
• The selection criteria look fine in general. But I do not understand how many projects in the starting set, how many projects were removed for each criterion in order to arrive at the final set of 770 projects. Please clearly explain step by step. (Add details about initial set of projects)

Reviewer 2:

• IMHO, the novelty of this work is marginal. I did not learn much from the paper as the findings are very similar to the prior works, i.e., the process metrics are the best and the experimental method is pretty standard. The only interesting part of this work is comparing the findings derived from a small vs large dataset, but still is not so well executed
• Despite the fair novelty, the topic addressed in this work is very significant as defect prediction and other predictive models are widely used in SE research and the size of the dataset could potentially impact the findings and results. The problem is well-motivated, where the authors show that many papers use a small number of datasets. This should highlight the generalizability threat of empirical studies.
• The authors describe that they aim to revisit the work of Rahman and Devanbu. Although the authors emphasize that they did not want to do an exact replication, this experimental design is too far different from the work of Rahman and Devanbu. More specifically, Rahman and Devanbu investigate this problem in the context of release-level defect prediction (i.e., predicting which file will be defective after the software is released). On the other hand, this work is more like a change-level defect prediction (i.e., which change will induce fix). Hence, the context is different. Consequently, the studied metrics and the ways that these metrics calculated are quite different. Hence, IMHO, the authors should clearly situate the work in the context of the change-level defect predict, which is unlike the work of Rahman and Devanbu.
• the paper does not provide an overview of the collected dataset (e.g., LOC, #Files, #PR, #Dev, Defective Ratio) (Added LOC, #Files, #Dev, Defective Ratio)
  • Since this work is not generally comparable with the work of Rahman and Devanbu (as per my comment 1), it would be better to conduct an experiment RQ3 for all the RQs. More specifically, the authors should run the experiment on a small dataset and check whether the process metrics are still the best in both small and large dataset in this context of a change-level defect prediction.
• While I did like the study approach of RQ3, I’m still concerned about its comparison. More specifically, the authors compare the metric importance between the Random forest model that is built in-the-large, and the Logistic Regression model that is built in-the-small. Why different classification techniques?
  • Q3 investigates the metric importance, the highly correlated metrics should be removed in order to achieve a stable conclusion
• It would be nice if the authors also conduct a statistical analysis.
• I’m very confused by the results of RQ4. In the introduction, the authors describe that product metrics are harder to compute than the process metrics. However, in the RQ4 results, the authors present in the other way around. So, I’m not sure which one is the correct results. Also, the results briefly are reported. It is better to describe the average computation for each project for process metrics and product metrics.

Reviewer 3:

• Authors said in part “Defective Commits step (2)” that you used Commit_Guru to identify commits. However I think the the classification of buggy commits made by Commit_Guru is based on a simple list of keywords. With this method they might miss a number of commit messages containing complex sentences without the Commit_Guru keywords. Automatic language processing and bug triage method based on classifiers could give more relevant results.
• Authors said that thick slice release-based is used to generate the train/test data with 60% of the first history as training set and the next 40% for the test. But this raises a question: what is the impact on the predictor quality to choose the 60% of the first history with a project containing only few hundred commits and/or with only one or two release(s)? Moreover why the choice 60/40 and not 80/20 compared another configuration of train/test?
• Evaluation criteria are relevant and explained. It is appreciable to have other criteria than classics recall, precision and ROC, all the more so Ifa and Popt20 are clearly applicable to predictors in software engineering. However, the choice could be more justified by authors.
• In RQ3, authors give the metric importance for Random Forest and Logistic Regression but what about the other two predictors (e.g., Naive Bayes and SVM) used in the study? For example authors could give the ranked coefficients for the SVM and the ranked probabilities for the Naive Bayes to have a point of comparison. Is it because Random Forests use a majority vote? A numerical comparison with predictors built on small scale of software projects might have been interesting and relevant.

[Suvodeep90]

Revisiting Common Bug Prediction Findings Using Effort-Aware Models

Kamei et al RQ1) Are process metrics still more effective than product metrics in effort-aware models? RQ2) Are package-level predictions still more effective than file-level predictions?

[Suvodeep90]

Experiments -

• Trying to replicate release based prediction on a smaller subset where it is possible.
• Package level prediction
• Correlation between metrics

[Suvodeep90]

• https://hal.archives-ouvertes.fr/hal-01433069/file/main.pdf
• https://arxiv.org/pdf/1801.10271.pdf
• https://dl.acm.org/doi/pdf/10.1145/2639490.2639505?casa_token=gvm90eF9S6YAAAAA:nAnCihYG8mtYwWly93fqNSsG59nme7WkP0blCcfNEsYMeHExLjeMYdbQ0cOfix3l8PWWZ4PqDV8tgQ
• https://www.lucapascarella.com/articles/2018/Pascarella_JSS_2018.pdf
• https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8169980&casa_token=6Jct53WbSDUAAAAA:mY1SYEICeoevGu917v9_jz74uq9Uk_PGqTvlqN9G5ug1ocOx0uGez9qtstgjPsNsNV07BOk3-w&tag=1

[Suvodeep90]

release based prediction (Set 1)

Process + Product:

• precision : 0.94
• recall : 0.99
• pf : 0.02
• f1 : 0.93
• g : 0.98
• auc : 0.9593753437792982
• pci_20 : 0.67
• ifa : 0.0

Process

• precision : 0.95
• recall : 0.99
• pf : 0.02
• f1 : 0.93
• g : 0.97
• auc : 0.9559504139025212
• pci_20 : 0.69
• ifa : 0.0

Product

• precision : 0.38
• recall : 0.47
• pf : 0.28
• f1 : 0.375
• g : 0.5
• auc : 0.5644070557647745
• pci_20 : 0.67
• ifa : 50.0

[Suvodeep90]

• RQ 1: In release based prediction settings, how do the process and code metrics compare to predict defect locations?
• RQ 2: Are process metrics more/less stable than code metrics?
• RQ 3: Are process metrics more/less portable than code metrics?
• RQ 4: Are process metrics more/less static than code metrics?
• RQ 5: Do models built from different sets of metrics stagnate across releases?
• RQ 6: Do stagnant models (based on stagnant metrics) tend to predict recurringly defective entities?

[Suvodeep90]

RQ 4: Are process metrics more/less static than code metrics?

In both cases the process metrics correlation is significantly better than other 2

Released based setting

JIT based setting

[Suvodeep90]

RQ 2: Are process metrics more/less stable than code metrics?

• The Statistical results shows there performance across releases are not statistically significant in all metrics except Popt_20.

[Suvodeep90]

RQ 1: In release based prediction settings, how do the process and code metrics compare to predict defect locations?

[Suvodeep90]

RQ 5: Do models built from different sets of metrics stagnate across releases?

We do not see very high correlation in process vs product vs process+product models. Although models based on product metrics are more stagnant then models based process metrics, the difference is not that stark when evaluating on 700 projects.

Metrics	correlation value	p -value
process	0.24	<0.001
product	0.53	<0.001
process+product	0.26	<0.001

[Suvodeep90]

RQ 6: Do stagnant models (based on stagnant metrics) tend to predict recurringly defective entities?

[Suvodeep90]

RQ2) Are package-level predictions still more effective than file-level predictions? (Kamei et al.)

[Suvodeep90]

completed