SLR update

[azhe825]

Motivation:

As in RQ6, the need of continuous updating SLR is identified in:

Carver, Jeffrey C., Edgar Hassler, Elis Hernandes, and Nicholas A. Kraft. "Identifying barriers to the systematic literature review process." In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 203-212. IEEE, 2013.

Another possible need is to reuse SLR, also identified in the above paper. The goal is to speed up the random selection process for one SLR by applying a model from another similar SLR, e. g., apply the model from Hall to Wahono to speed up the initial seed collection.

Two goals, two solutions

• GOAL1: update SLR: review protocal stays the same, data (top N important terms from feature extraction) can be different.
  • SOLUTION1: continuously update the previous model and apply it to new data.
• GOAL2: reuse SLR: apply model to a similar SLR. Review protocal stays similar but different, data as well.
  • SOLUTION2: apply the old model only to replace random selection, once got a initial training set, start a new model, purely based on the new data.

[azhe825]

Result

ONE repeat result. Each one stop at 90% retrieval rate.

update SLR

	Hall <= 2007	Hall > 2007
Initial Size	4846	4066
Final Size	55	51

• Hall <= 2007, HCCA.

• Hall > 2007, add in new data, re-featurize data set (Hall <= 2007 + Hall > 2007). Use the training examples from Hall <= 2007 to retrain a model and continue certainty sampling with aggressive undersampling on Hall > 2007 data only.

reuse SLR

• extract the model (w,b) from Hall > 2007, transform it to fit the feature vector of Wahono. Apply the transformed model to predict for "relevant" studies in Wahono, replacing random sampling. Once get ONE relevant" study retrieved, start usual HCCA.

So reuse may not be a great idea.

Problem

• Is the result stable?
• If necessary, how to represent the N repeats result?

[azhe825]

25 Repeat Result

update SLR

	Hall <= 2007	Hall > 2007
Initial Size	4846	4066
Final Size	55	51

• Hall <= 2007, HCCA.

• Hall > 2007, add in new data, re-featurize data set (Hall <= 2007 + Hall > 2007). Use the training examples from Hall <= 2007 to retrain a model and continue certainty sampling with aggressive undersampling on Hall > 2007 data only.

Continuous updating:

Start from beginning:

reuse SLR

• extract the model (w,b) from Hall > 2007, transform it to fit the feature vector of Wahono. Apply the transformed model to predict for "relevant" studies in Wahono, replacing random sampling. Once get ONE relevant" study retrieved, start usual HCCA.

Reuse model:

Start from beginning:

Conclusion

UPDATE has great result. REUSE can reduce variance.

[nkraft]

Relevant, recent work: http://dx.doi.org/10.1145/2961111.2962630

[azhe825]

will include the paper for reference, a great one to justify the need to update SLR.

[nkraft]

Not sure if this is the best spot to put this, but: http://www.win.tue.nl/~aserebre/SCAM2016.pdf

Discusses the need for both SLR and snowballing

[azhe825]

This is cool~