search

ai-se / weeklies

0 stars 0 forks source link

10000+, April 9 #9

Open Suvodeep90 opened 4 years ago

Suvodeep90 commented 4 years ago

What has been completed(Done) 1) Tried with 3 different strategies for reducing the number of commits required for the collection. 2) Compare the metric results for process vs product vs process+product

What are you working on(Doing) 1) Literature reviews on process vs product metrics 2) Tring to see what are the existing hypothesis and how they change with 1000 projects 3) Data collection 800 projects being collected this week. 4) Collecting release information for the 1000 java project I am working with. 5) Other ways to reduce the number of commits we are mining.

What you will be working on(To Do) 1) Review the GENERAL paper comments. 2) Data Collection 3) Getting more clusters to speed up data collection

Any Roadblocks

Additional context

Suvodeep90 commented 4 years ago

Results for selecting a subset of commits based on process data to explore the product data

Strategy_0:

Cluster all commits and then randomly Pick 100 commits. Didn't give good results. Will add the results to the stats test.

Strategy_1:

For Each file, that is buggy pick one defective commit to analyzing further. Reduction 52%

Strategy_2:

Select all files which have only two defective commits, for rest pick the most different ones. By selecting the commits which are furthest apart. Reduction 35%

Strategy_3:

Pick the files which have been defective more than the median number of defects per file and then only choose 1 at random (same as strategy 1). Reduction 71%

Strategy_4:

Pick the files which have been defective more than the median number of defects per file and then only choose two most different ones (same as strategy 2). Reduction 60%

Precision
   1 ,                    All_Commits ,    0.71   , 0.21 (               |    --*-      ), 0.48,  0.68,  0.76,  0.82,  0.89
   1 ,                     Strategy_2 ,    0.72   , 0.21 (               |    ---*      ), 0.50,  0.69,  0.77,  0.83,  0.90
   1 ,                     Strategy_1 ,    0.72   , 0.21 (               |    ---*-     ), 0.50,  0.70,  0.78,  0.84,  0.91
   1 ,                     Strategy_3 ,    0.73   , 0.22 (               |    ---*-     ), 0.50,  0.70,  0.79,  0.85,  0.91
   1 ,                     Strategy_4 ,    0.76   , 0.21 (               |      -*-     ), 0.57,  0.74,  0.80,  0.85,  0.93

Recall
   1 ,                    All_Commits ,    0.7   , 0.23 (               |   ---*-      ), 0.38,  0.66,  0.76,  0.82,  0.91
   1 ,                     Strategy_2 ,    0.71   , 0.23 (               |    ---*-     ), 0.43,  0.68,  0.78,  0.84,  0.91
   1 ,                     Strategy_1 ,    0.72   , 0.23 (               |    ---*-     ), 0.43,  0.68,  0.79,  0.85,  0.93
   1 ,                     Strategy_3 ,    0.73   , 0.23 (               |    ---*--    ), 0.40,  0.69,  0.80,  0.87,  0.93
   1 ,                     Strategy_4 ,    0.74   , 0.23 (               |     ---*-    ), 0.49,  0.72,  0.81,  0.87,  0.94

Pf
   1 ,                     Strategy_4 ,    0.17   , 0.15 (  --*          |              ), 0.00,  0.09,  0.14,  0.20,  0.34
   1 ,                    All_Commits ,    0.16   , 0.13 (  --*          |              ), 0.03,  0.09,  0.14,  0.18,  0.29
   1 ,                     Strategy_2 ,    0.17   , 0.14 (  --*          |              ), 0.03,  0.10,  0.15,  0.20,  0.33
   1 ,                     Strategy_1 ,    0.18   , 0.14 (  --*-         |              ), 0.03,  0.10,  0.16,  0.21,  0.33
   1 ,                     Strategy_3 ,    0.19   , 0.16 (  --*-         |              ), 0.03,  0.10,  0.16,  0.22,  0.38

Popt_20
   1 ,                     Strategy_4 ,    0.63   , 0.21 (               | --*-         ), 0.38,  0.58,  0.66,  0.73,  0.86
   1 ,                     Strategy_3 ,    0.64   , 0.22 (               | ---*-        ), 0.42,  0.59,  0.67,  0.74,  0.88
   1 ,                     Strategy_1 ,    0.66   , 0.21 (               |  --*-        ), 0.43,  0.62,  0.69,  0.76,  0.88
   1 ,                     Strategy_2 ,    0.66   , 0.21 (               |  --*--       ), 0.43,  0.62,  0.69,  0.77,  0.89
   1 ,                    All_Commits ,    0.68   , 0.22 (               |  ---*-       ), 0.44,  0.63,  0.71,  0.79,  0.91

IFA
   1 ,                     Strategy_4 ,    26.02   , 508.76 (*              |              ), 0.00,  0.00,  0.00,  1.00,  2.00
   1 ,                    All_Commits ,    10.21   , 105.28 (*              |              ), 0.00,  0.00,  0.00,  1.00,  3.00
   1 ,                     Strategy_1 ,    10.0   , 105.3 (*              |              ), 0.00,  0.00,  0.00,  1.00,  2.00
   1 ,                     Strategy_2 ,    10.22   , 105.42 (*              |              ), 0.00,  0.00,  0.00,  1.00,  2.00
   1 ,                     Strategy_3 ,    25.99   , 513.89 (*              |              ), 0.00,  0.00,  0.00,  1.00,  2.00

AUC
   1 ,                    All_Commits ,    0.75   , 0.17 (               |      -*      ), 0.60,  0.74,  0.79,  0.83,  0.89
   1 ,                     Strategy_3 ,    0.75   , 0.17 (               |     --*-     ), 0.60,  0.73,  0.79,  0.84,  0.90
   1 ,                     Strategy_2 ,    0.75   , 0.17 (               |      -*      ), 0.61,  0.74,  0.79,  0.83,  0.88
   1 ,                     Strategy_1 ,    0.76   , 0.17 (               |      -*-     ), 0.61,  0.74,  0.79,  0.84,  0.89
   1 ,                     Strategy_4 ,    0.77   , 0.18 (               |      --*     ), 0.62,  0.75,  0.81,  0.84,  0.91

F1
   1 ,                    All_Commits ,    0.69   , 0.21 (               |    --*-      ), 0.43,  0.67,  0.75,  0.81,  0.88
   1 ,                     Strategy_2 ,    0.71   , 0.21 (               |    ---*      ), 0.45,  0.67,  0.77,  0.83,  0.88
   1 ,                     Strategy_1 ,    0.71   , 0.21 (               |    ---*      ), 0.45,  0.69,  0.78,  0.83,  0.89
   1 ,                     Strategy_3 ,    0.72   , 0.22 (               |    ---*-     ), 0.44,  0.69,  0.79,  0.85,  0.90
   1 ,                     Strategy_4 ,    0.74   , 0.21 (               |     --*-     ), 0.53,  0.73,  0.80,  0.85,  0.91

G-Score
   1 ,                    All_Commits ,    0.72   , 0.18 (               |    ---*      ), 0.50,  0.69,  0.77,  0.82,  0.89
   1 ,                     Strategy_2 ,    0.73   , 0.18 (               |     --*      ), 0.51,  0.71,  0.78,  0.83,  0.89
   1 ,                     Strategy_1 ,    0.74   , 0.18 (               |     --*-     ), 0.51,  0.72,  0.79,  0.84,  0.90
   1 ,                     Strategy_3 ,    0.74   , 0.19 (               |     --*-     ), 0.49,  0.71,  0.79,  0.84,  0.91
   1 ,                     Strategy_4 ,    0.76   , 0.18 (               |      -*-     ), 0.56,  0.74,  0.80,  0.85,  0.91
Suvodeep90 commented 4 years ago

Results for Process vs Product vs Process+Product

Precision
   1 ,                        Product ,    0.5   , 0.21 (            ---*--            ), 0.22,  0.42,  0.51,  0.61,  0.73
   1 ,                        Process ,    0.48   , 0.24 (           ----*--            ), 0.00,  0.39,  0.52,  0.62,  0.73
   2 ,                Process+Product ,    0.71   , 0.21 (               |    --*-      ), 0.48,  0.68,  0.76,  0.82,  0.89

Recall
   1 ,                        Process ,    0.46   , 0.25 (          ----*|--            ), 0.00,  0.34,  0.50,  0.61,  0.77
   1 ,                        Product ,    0.49   , 0.23 (           ---*|--            ), 0.18,  0.38,  0.50,  0.62,  0.77
   2 ,                Process+Product ,    0.7   , 0.23 (               |   ---*-      ), 0.38,  0.66,  0.76,  0.82,  0.91

Pf
   1 ,                        Process ,    0.1   , 0.06 ( -*            |              ), 0.02,  0.06,  0.09,  0.13,  0.18
   2 ,                Process+Product ,    0.16   , 0.13 (  --*          |              ), 0.03,  0.09,  0.14,  0.18,  0.29
   3 ,                        Product ,    0.28   , 0.17 (     --*--     |              ), 0.08,  0.18,  0.26,  0.34,  0.50

Popt_20
   1 ,                Process+Product ,    0.68   , 0.22 (               |  ---*-       ), 0.44,  0.63,  0.71,  0.79,  0.91
   1 ,                        Product ,    0.69   , 0.23 (               |   --*--      ), 0.43,  0.64,  0.72,  0.81,  0.93
   2 ,                        Process ,    0.79   , 0.25 (               |      ----*-  ), 0.50,  0.76,  0.88,  0.94,  0.98

IFA
   1 ,                Process+Product ,    10.21   , 105.28 (*              |              ), 0.00,  0.00,  0.00,  1.00,  3.00
   1 ,                        Product ,    28.69   , 513.65 (*              |              ), 0.00,  0.00,  1.00,  2.00,  6.00
   1 ,                        Process ,    21.36   , 224.98 (*              |              ), 0.00,  1.00,  2.00,  3.00, 15.00

AUC
   1 ,                        Product ,    0.59   , 0.14 (               |-*-           ), 0.49,  0.56,  0.60,  0.64,  0.74
   2 ,                        Process ,    0.67   , 0.14 (               |  --*-        ), 0.50,  0.63,  0.69,  0.74,  0.81
   3 ,                Process+Product ,    0.75   , 0.17 (               |      -*      ), 0.60,  0.74,  0.79,  0.83,  0.89

F1
   1 ,                        Process ,    0.46   , 0.23 (          ----*|-             ), 0.00,  0.36,  0.50,  0.60,  0.72
   1 ,                        Product ,    0.48   , 0.21 (           ---*|--            ), 0.21,  0.39,  0.50,  0.61,  0.73
   2 ,                Process+Product ,    0.69   , 0.21 (               |    --*-      ), 0.43,  0.67,  0.75,  0.81,  0.88

G-Score
   1 ,                        Process ,    0.5   , 0.24 (            ---|*--           ), 0.09,  0.41,  0.54,  0.65,  0.78
   1 ,                        Product ,    0.51   , 0.19 (            ---|*-            ), 0.25,  0.43,  0.54,  0.62,  0.73
   2 ,                Process+Product ,    0.72   , 0.18 (               |    ---*      ), 0.50,  0.69,  0.77,  0.82,  0.89
Suvodeep90 commented 4 years ago

RQs in metric related papers

How, and Why, Process Metrics Are Better

Metrics used: process, product, process+Size, process+product+Size

  1. In release-based prediction settings, how do the process and code metrics compare to predict defect locations? Checked different models with different metrics types showed they perform similarly for different models.

  2. Are process metrics more/less stable than code metrics? Check different metrics performance stability across 5 test releases.

  3. Are process metrics more/less portable than code metrics? Check the performance difference between different types of metrics within and cross-project defect prediction.

  4. Are process metrics more/less static than code metrics? correlation of metric values between 2 releases.

  5. Do models built from different sets of metrics stagnate across releases? Checking if the same modules are being targeted as defective although they are not.

  6. In our results, we see process + product do give us better results in contrary to Dr. Devenvu's result.

A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction

  1. Are change metrics more useful in detecting defective source code than code metrics?

  2. Which change metrics are good defect predictors?

  3. How accurate are those models?

  4. How much does it cost a software organization to utilize defect prediction models and what are the benefits?

Mining Metrics to Predict Component Failures

  1. Do complexity metrics correlate with failures in the field? For each project, we can find a set of complexity metrics that correlate with post-release defects.

  2. Is there a single set of metrics that predicts post-release defects in all projects? No

  3. Can we combine metrics to predict post-release defects? Predictors obtained from the principal component analysis are useful in building regression models to estimate post-release defects.

  4. Are predictors obtained from one project applicable to other projects? Predictors are accurate only when obtained from the same or similar projects.

Using Historical In-Process and Product Metrics for Early Estimation of Software Failures

Similar RQs as the last one

Which process metrics can significantly improve defect prediction models? An empirical study

Checked different process metrics to identify, which one is better for predicting defects.

An Empirical Study on Software Defect Prediction with a Simplified Metric Set

Tried to predict defects with reduced attributes with feature selectors.

  1. Which type of defect prediction model is more suitable for a specific project between WPDP and CPDP? Are there suitable models specific to WPDP and CPDP.

  2. Does the predictor build with a simplified metric set work well? It doesn't seem too much improvement is there.

  3. Which classifier is more likely to be the choice of defect prediction with a simplified metric set? They say NB.

  4. Is there a minimum metric subset that facilitates the procedure for general defect prediction? The paper has shown a subset.