literature reivew ---zhe

[timm]

are your slides somewhere everyone can see them?

[azhe825]

Slides: https://docs.google.com/a/ncsu.edu/presentation/d/1GPGqyWrzpjaEdTc1D3GTgqQUiVUEJYaC6wKKzDuCrIQ/edit?usp=sharing

But... I don't think this slide is understandable by just read it...

[timm]

please place pdf in a public space and ping the group

[azhe825]

I have updated the slides. Where should I put it? Google drive is not suitable? All NCSU people can see (but not edit) it on google drive.

[timm]

pdf in github is fine. fyi- github supports pdf preview

[azhe825]

Seiffert, C., Khoshgoftaar, T. M., & Van Hulse, J. (2009). Improving software-quality predictions with data sampling and boosting. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 39(6), 1283-1294. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5233804

Oversampling methods: RUS, ROS, SM, BSM, and WE. As well as boosting to tackle imbalance.

Performance metrics: K-S statistic and AUC.

[azhe825]

SMOTE + text mining

• Borderline-SMOTE
  • Han, H., Wang, W. Y., & Mao, B. H. (2005). Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. In Advances in intelligent computing (pp. 878-887). Springer Berlin Heidelberg.
  • http://link.springer.com/chapter/10.1007/11538059_91
  • Only over-sample the borderline.
  • Interesting concept: 2 different types of imbalance -- between-class and within-class.
• SMOTE + SVM on text mining
  • Sun, A., Lim, E. P., & Liu, Y. (2009). On strategies for imbalanced text classification using SVM: A comparative study. Decision Support Systems, 48(1), 191-201.
  • http://www.sciencedirect.com/science/article/pii/S0167923609001754
  • Data sets: 20-Newsgroups, Reuters-21578, and WebKB
  • Linear SVM + SRAND or CLUS or SMOTE
  • Conclusion is SVM alone is the best... which means SVM is better than SVM + SMOTE...
  • Some tests have been done by changing the number of neighbors SMOTE uses, no difference found.
• A more recent case. Focusing on feature selection.
  • Chang, F., Guo, J., Xu, W., & Yao, K. (2015). A Feature Selection Method to Handle Imbalanced Data in Text Classification. Journal of Digital Information Management, 13(3), 169.
  • http://dline.info/fpaper/jdim/v13i3/v13i3_6.pdf
  • Resampling: SMOTE
  • Classifier: linear SVM
  • Evaluation: Macro F1--which is our F1_M
  • Feature Selection: CHI, MI, NFS.
  • Basic ideas of these feature selection methods: all consider number of docs containing token_i and belongs (not belongs) to class_j. Then some score with similar idea to tfidf, but with these in (out of) class_i numbers.
  • Data set: Reuters-21578
  • Result: Macro F1 reaches 0.85 with 1000 features. SVM+SMOTE+CHI and SVM+SMOTE+NFS

SMOTE + Software Engineering (defect prediction appears a lot)

• SMOTE on defect prediction
  • Pelayo, Lourdes, and Scott Dick. "Applying novel resampling strategies to software defect prediction." Fuzzy Information Processing Society, 2007. NAFIPS'07. Annual Meeting of the North American. IEEE, 2007.
  • http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4271036
  • Data sets: NASA projects: CM1, KC1, KC2 and PC1
  • SMOTE to total balance will not give the best accuracy.
  • 300% SMOTE gives the best accuracy.
• SMOTE on software quality classification, Class noise considered
  • Van Hulse, J., & Khoshgoftaar, T. (2009). Knowledge discovery from imbalanced and noisy data. Data & Knowledge Engineering, 68(12), 1513-1542.
  • http://www.sciencedirect.com/science/article/pii/S0169023X09001141
  • Classifier: 11 including SVM, Naive Bayes, C4.5, ANN
  • Resampling: 7 including SMOTE and borderline-SMOTE
  • Evaluation: AUC and Kolmogorov–Smirnov statistic (KS)
  • Data sets: CM1, MW1, PC1, KC1, and KC3
  • Class noise severely impacts the performance.
  • For me it seems that SVM is among the best.
  • But the authors' conclusion is that simple algorithms like Naive Bayes and KNN are more robust than complicate ones like SVM
• More recent, review
  • Reformat, M. (2014). Special section: Applications of computational intelligence and machine learning to software engineering. Information Sciences, 259, 393-395.
  • http://www.sciencedirect.com/science/article/pii/S0020025513008207

Text mining + Software Engineering (Stackoverflow)

• See https://github.com/ai-se/SMOTE/blob/master/pdf/SE%20text%20mining%20in%20StackExchange.pdf

Text + SE + SMOTE

• Our works will lie here