Zhe's milestone (undetermined)

[azhe825]

Learning based Systematic Literature Review (For Ph.d)

Current Baseline Approach: Searching + filtering, then linear review.

• Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, Stephen Linkman, Systematic literature reviews in software engineering – A systematic literature review, Information and Software Technology, Volume 51, Issue 1, January 2009, Pages 7-15, ISSN 0950-5849
• Kitchenham, Barbara Ann, David Budgen, and Pearl Brereton. Evidence-Based Software Engineering and Systematic Reviews. Vol. 4. CRC Press, 2015.
• Marshall, Christopher, Pearl Brereton, and Barbara Kitchenham. "Tools to support systematic reviews in software engineering: a cross-domain survey using semi-structured interviews." In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, p. 26. ACM, 2015.
• Souza, Draylson M., Katia R. Felizardo, and Ellen F. Barbosa. "A Systematic Literature Review of Assessment Tools for Programming Assignments." In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET), pp. 147-156. IEEE, 2016.

Challenges:

1. Reduce review cost
2. Imbalance (target class is always minority)
3. Utilize knowledge from users

To Do:

• [ ] Work bench
  • [x] get data (linear review, time recorded --- baseline approach)
  • [ ] construct citation matrix
• [x] Basic method
  • searching (Elasticsearch) [1, 2, 3]
  • learning (lexical analysis with term frequency, l2 normalization, SVM...)
  • active learning (uncertainty sampling, certainty sampling) [1, 2]
  • data balancing? need to test. [2]
• [ ] Obtain a better initial training set for active learning with fewer reviews [1, 2]
  • baseline: random sampling
  • clustering based on lexical analysis [1, 2]
  • spectral clustering based on citation matrix (citemap?) (Possible expansion: utilize citation matrix on learning) [1, 2]
• [ ] Get user involved [3]
  • show important features and let user to mask them [3]
  • allow user to re-review important documents (support vectors) [3]
  • let user explore the clusters [3]
• [ ] Visualization of Learning Result (not sure how this can help right now) [1, 3]
  • pretty graphs (d3.js, kibana)
  • present results in each cluster

[azhe825]

Implementations of machine assisted reading

1. (Multi-objective) optimization: reduce number of evaluations by ranking candidates. Ongoing
2. Defect prediction:
   • standard machine assisted reading on projects with no labeled data. (test->train->rank->test->train->rank->...)
   • updating with machine assisted reading on new version of codes
   • reuse with machine assisted reading on similar projects

[timm]

Lets talk to this next time we meet

[azhe825]

Sure