Open Suvodeep90 opened 5 years ago
Baseline results: Out of 1385 projects, I selected the ones with more than 50 rows in the dataset, that gave us 711 projects. Some projects had some issues. Final dataset has 633 projects.
We applied LR(9 parameters tuned using DE). DE was tuned for F1 measure. And the chart shows f1-measure for individual projects.
In this experiment - each project was first divided into a train(67%) and test set(33%), then the train data was further divided into train and tune set with 5 fold cross-validation. The model was trained on 80% of the data and tuned on 20%. Then the best model was tested using the test set.
looks very nice
Out of 1385 projects, I selected the ones with more than 50 rows in the dataset, that gave us 711 projects. Some projects had some issues. Final dataset has 633 projects.
Default LR on the dataset: The Data was divided into Train and test set using a 5 fold cross validation with 5 repeats. There is no FSS, no Smote and use of default parameter setting for LR from sklearn.
The results are shown here are median score of 25 runs on each project. The meaures contain - precision,recall, f1-score, d2h, g-score, pci20, ifa. https://docs.google.com/spreadsheets/d/1uwN4dI-kpkparZ5tj15yl7_jwwaK8lDDIU0-zc9qgyM/edit?usp=sharing
Median of Median Results: F1:0.69 Prec:0.69 Recall:0.71 g-score:0.46 d2h:0.72 pci20:95 ifa:5
10 Random Project tuned with DE for f1-measure results - With FSS as CFS, Smotuned(tuned for 3 parameters), LR tuned for 10 hyper-parameters. with 5*5 cross-validation. https://docs.google.com/spreadsheets/d/1hVMbGE4lUvWWlH5nFna5Dvyy0nLDF2nmzqHz4rOdS4Q/edit?usp=sharing
Here hyper-parameter optimizer is not helping. Investigating the reason.
'TLOC', 'TNF', 'TNC', 'TND', 'LOC', 'CL', 'NStmt', 'NFunc', 'RCC', 'MNL', 'avg_WMC', 'max_WMC', 'total_WMC', 'avg_DIT', 'max_DIT', 'total_DIT', 'avg_RFC', 'max_RFC', 'total_RFC', 'avg_NOC', 'max_NOC', 'total_NOC', 'avg_CBO', 'max_CBO', 'total_CBO', 'avg_DIT.1', 'max_DIT.1', 'total_DIT.1', 'avg_NIV', 'max_NIV', 'total_NIV', 'avg_NIM', 'max_NIM', 'total_NIM', 'avg_NOM', 'max_NOM', 'total_NOM', 'avg_NPBM', 'max_NPBM', 'total_NPBM', 'avg_NPM', 'max_NPM', 'total_NPM', 'avg_NPRM', 'max_NPRM', 'total_NPRM', 'avg_CC', 'max_CC', 'total_CC', 'avg_FANIN', 'max_FANIN', 'total_FANIN', 'avg_FANOUT', 'max_FANOUT', 'total_FANOUT', 'NRev', 'NFix', 'avg_AddedLOC', 'max_AddedLOC', 'total_AddedLOC', 'avg_DeletedLOC', 'max_DeletedLOC', 'total_DeletedLOC', 'avg_ModifiedLOC', 'max_ModifiedLOC', 'total_ModifiedLOC','Buggy'
i feel like we are closing in on a methodology, a check list of sorts for large scale data mining. e.g. if we started building a library of sanity check demons as per for first point then we'd have a product to offer other people
please, for the 1385 can you clarify the following:
there are some wondrous things about the results:
there's something wrong with your options.
lami and ambidata has the expected effect (DE improves things). the other datasets do not see that improve
but the other data sets show no improvement.
so:
This new results are from the new dataset, i think the data quality is better than the commit guru dataset. The same rig is being applied to the both dataset.
I am not exactly sure why this is happening. But i am rechecking if there is some issue or not. In the mean time i will train for pd. I have not tried to optimize DE with dom score before, but i know the concept, i am trying to code that up.
Also I will update the answers to the last comment shortly.
if it essentially the same rig as used for CommitGuru150 then the source of this quick might be in the data, not your algorithms. the performance here is amazingly good. did u read the paper this data came from? (*) all the careful human curation? this data set could be somewhat exceptional
(*) BTW, applying the rule of KNOW THY DATA, that is a requirement, not. a question.
The original dataset was collected by Dr. Mockus from SourceForge and GoogleCode.
suvodeep comments that this data was severely pruned prior to analysus
the results here seen high w.r.t other results we have seen in other data sets. this could either be a selection bias here (but the sanity checks seem sane) or it could be the result of poor pre-processing in other data sets.
could you edit https://github.com/ai-se/bellwether_community/issues/14#issuecomment-508883462 and make sure all the 13 points are addressed?
And LR is fast! and polynomial SVM is slow
local Lib = require("lib")
local Burn = require("burn")
local Learn = require("learn")
local Object = require("object")
Nb = {}
Nb = Object:new{maybes=0, name="nb"}
--------- --------- --------- --------- --------- ---------
function Nb:new()
local n=Object.new(self)
n.datas={}
return n
end
function Nb:like(cells, data1, m,k, n, inc)
local prior= ( #data1.rows + k ) / ( n+ k*self.maybes )
local like = math.log(prior)
for _,col in pairs(data1.x.cols) do
local x = cells[ col.pos ]
if x ~= Burn.ignore then
if col:nump() then
inc = Lib.normal(x, col.mu, col:sd())
else
inc = ( (col.counts or 0) + m * prior ) /
( #data1.rows + m )
end
like = like + math.log(inc) end end
return like
end
function Nb:what(cells,data0)
local c = cells[ data0._class.pos ]
if not self.datas[c] then
self.maybes = self.maybes + 1
self.datas[c] = data0:clone()
end
return self.datas[c]
end
function Nb:inc(cells,data0)
local data1 = self:what(cells,data0)
data1:inc(cells)
data0:inc(cells)
end
function Nb:predict(cells,data0)
local max,got = -10^32,nil
for h1,data1 in pairs(self.datas) do
got = got or h1
local l = self:like(cells, data1, Burn.nb.m,
Burn.nb.k, #data0.rows)
if l > max then max,got = l,h1 end
end
return got
end
function Nb:report(era,goal,log)
Learn.report(era,log,self.name,goal)
end
return Nb
http://menzies.us/pdf/12localb.pdf
hey... what's the above new charts?
these are the new charts with updated metric calculations .. I just wanted to store them somewhere. not ready for reporting yet..
Time:
160 projects: - 73053 secs
Hierarchical Clustering results:
[cluster_id=0] N_children: 9 N_samples: 697
[cluster_id=1] N_children: 9 N_samples: 87
[cluster_id=2] N_children: 0 N_samples: 2
[cluster_id=3] N_children: 0 N_samples: 7
[cluster_id=4] N_children: 0 N_samples: 17
[cluster_id=5] N_children: 0 N_samples: 6
[cluster_id=6] N_children: 0 N_samples: 18
[cluster_id=7] N_children: 0 N_samples: 15
[cluster_id=8] N_children: 0 N_samples: 12
[cluster_id=9] N_children: 0 N_samples: 6
[cluster_id=10] N_children: 0 N_samples: 4
[cluster_id=11] N_children: 2 N_samples: 4
[cluster_id=12] N_children: 0 N_samples: 1
[cluster_id=13] N_children: 0 N_samples: 3
[cluster_id=14] N_children: 10 N_samples: 103
[cluster_id=15] N_children: 0 N_samples: 3
[cluster_id=16] N_children: 0 N_samples: 4
[cluster_id=17] N_children: 0 N_samples: 16
[cluster_id=18] N_children: 0 N_samples: 11
[cluster_id=19] N_children: 0 N_samples: 19
[cluster_id=20] N_children: 0 N_samples: 3
[cluster_id=21] N_children: 0 N_samples: 10
[cluster_id=22] N_children: 0 N_samples: 6
[cluster_id=23] N_children: 0 N_samples: 13
[cluster_id=24] N_children: 0 N_samples: 18
[cluster_id=25] N_children: 0 N_samples: 2 [cluster_id=26] N_children: 0 N_samples: 4 [cluster_id=27] N_children: 0 N_samples: 1 [cluster_id=28] N_children: 6 N_samples: 64
[cluster_id=29] N_children: 0 N_samples: 9
[cluster_id=30] N_children: 0 N_samples: 12
[cluster_id=31] N_children: 0 N_samples: 15
[cluster_id=32] N_children: 0 N_samples: 16
[cluster_id=33] N_children: 0 N_samples: 5
[cluster_id=34] N_children: 0 N_samples: 7
[cluster_id=35] N_children: 19 N_samples: 222
[cluster_id=36] N_children: 0 N_samples: 12
[cluster_id=37] N_children: 0 N_samples: 10
[cluster_id=38] N_children: 0 N_samples: 4
[cluster_id=39] N_children: 0 N_samples: 19
[cluster_id=40] N_children: 0 N_samples: 20
[cluster_id=41] N_children: 0 N_samples: 11
[cluster_id=42] N_children: 0 N_samples: 16
[cluster_id=43] N_children: 0 N_samples: 8
[cluster_id=44] N_children: 0 N_samples: 13
[cluster_id=45] N_children: 0 N_samples: 7
[cluster_id=46] N_children: 0 N_samples: 17
[cluster_id=47] N_children: 0 N_samples: 19
[cluster_id=48] N_children: 0 N_samples: 6
[cluster_id=49] N_children: 0 N_samples: 5
[cluster_id=50] N_children: 0 N_samples: 12
[cluster_id=51] N_children: 0 N_samples: 13
[cluster_id=52] N_children: 0 N_samples: 19
[cluster_id=53] N_children: 0 N_samples: 4
[cluster_id=54] N_children: 0 N_samples: 7
[cluster_id=55] N_children: 20 N_samples: 210
[cluster_id=56] N_children: 0 N_samples: 11
[cluster_id=57] N_children: 0 N_samples: 18
[cluster_id=58] N_children: 0 N_samples: 3
[cluster_id=59] N_children: 0 N_samples: 13
[cluster_id=60] N_children: 0 N_samples: 9
[cluster_id=61] N_children: 0 N_samples: 18
[cluster_id=62] N_children: 0 N_samples: 15
[cluster_id=63] N_children: 0 N_samples: 2
[cluster_id=64] N_children: 0 N_samples: 5
[cluster_id=65] N_children: 0 N_samples: 13
[cluster_id=66] N_children: 0 N_samples: 6
[cluster_id=67] N_children: 0 N_samples: 11
[cluster_id=68] N_children: 0 N_samples: 2
[cluster_id=69] N_children: 0 N_samples: 1
[cluster_id=70] N_children: 0 N_samples: 11
[cluster_id=71] N_children: 0 N_samples: 18
[cluster_id=72] N_children: 0 N_samples: 18
[cluster_id=73] N_children: 0 N_samples: 20
[cluster_id=74] N_children: 0 N_samples: 7
[cluster_id=75] N_children: 0 N_samples: 9
Ideas:
To apply hyper-parameter optimization .. if we tune each project separately and apply and select the best model out of them to use it as bellwether.
Stats Test (compare with self test)
F1 score
Precision
Recall
g-score
pf
in each community(cluster) can we find bellwether using stats test, and the models who rank best are the bellwether?
we can if we want to be slow. but lets see what happens if we bellwether on steroids first
any news on 600+?
todo
expectations
The expectation from results: