How to start

[azhe825]

Core techniques

• UPDATE: load in a labeled set as training data
• PRESUME: sample unlabeled data as negative examples for training
• DISCARD: exclude loaded set (from UPDATE) when training the model
• Auto_Syn: generate an example 'relevant' study
• BM25: sort unlabeled studies by the relevance to a set of keywords (tf-idf based)
• RANDOM: random sample from the unlabeled set. works as a baseline

Scenarios

• 1. update (all) scenario: have data (relevant and irrelevant examples) for previous SLR, now need to include more studies
• 2. update (relevant) scenario: have data (relevant examples only) for previous SLR, now need to include more studies
• 3. reuse scenario: have data from other existing SLRs, now need to start a new SLR with similar topics.
• 4. fresh start scenario: no data for seeding

If one strategy can be applied to a scenario of rank R, it can be applied to all scenarios with rank <= R.

START strategies:

• UPDATE: 1. update (all)
• UPDATE+PRESUME: 2. update (relevant)
• REUSE = UPDATE+PRESUME+DISCARD: 3. reuse
• Auto_Syn + REUSE: 4. fresh
• BM25 +PRESUME: 4. fresh
• RANDOM: 4. fresh

Experiments:

Stop at 95% relevant found (knowing the true number)

update scenario:

Hall2007+.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,         BM25 ,    140.00  ,  10.00 (*              |              ), 140.00,  140.00,  150.00
   1 ,        REUSE ,    150.00  ,  10.00 (-*             |              ), 140.00,  150.00,  150.00
   2 ,     Auto_Syn ,    155.00  ,  20.00 (--*            |              ), 140.00,  160.00,  160.00
   3 ,   UPDATE_POS ,    160.00  ,  20.00 ( -*            |              ), 150.00,  160.00,  170.00
   3 ,   UPDATE_ALL ,    160.00  ,  0.00 (  *            |              ), 160.00,  160.00,  160.00
   4 ,       RANDOM ,    205.00  ,  140.00 (    ---*       |              ), 180.00,  210.00,  320.00

K_all3+.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,   UPDATE_POS ,    675.00  ,  160.00 (  ---*         |              ), 610.00,  690.00,  770.00
   2 ,     Auto_Syn ,    840.00  ,  0.00 (           *   |              ), 840.00,  840.00,  840.00
   3 ,        REUSE ,    1160.00  ,  0.00 (               |         *    ), 1160.00,  1160.00,  1160.00
   4 ,         BM25 ,    1190.00  ,  0.00 (               |          *   ), 1190.00,  1190.00,  1190.00
   5 ,       RANDOM ,    1210.00  ,  10.00 (               |           *  ), 1210.00,  1210.00,  1220.00

Danijel2005+.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,   UPDATE_ALL ,    230.00  ,  0.00 ( *             |              ), 230.00,  230.00,  230.00
   2 ,         BM25 ,    240.00  ,  0.00 (  *            |              ), 240.00,  240.00,  240.00
   3 ,   UPDATE_POS ,    300.00  ,  70.00 (  --*          |              ), 250.00,  300.00,  320.00
   4 ,       RANDOM ,    315.00  ,  110.00 (    -*         |              ), 280.00,  320.00,  390.00
   4 ,     Auto_Syn ,    330.00  ,  10.00 (      *        |              ), 330.00,  330.00,  340.00
   4 ,        REUSE ,    365.00  ,  60.00 (      --*      |              ), 340.00,  370.00,  400.00

Wahono2008+.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,         BM25 ,    260.00  ,  10.00 (  *            |              ), 260.00,  260.00,  270.00
   2 ,   UPDATE_POS ,    300.00  ,  20.00 (   *           |              ), 290.00,  300.00,  310.00
   3 ,   UPDATE_ALL ,    320.00  ,  0.00 (    *          |              ), 320.00,  320.00,  320.00
   4 ,       RANDOM ,    370.00  ,  130.00 (   ---*        |              ), 310.00,  370.00,  440.00
   4 ,        REUSE ,    395.00  ,  80.00 (     --*       |              ), 360.00,  400.00,  440.00
   5 ,     Auto_Syn ,    1030.00  ,  0.00 (               |             *), 1030.00,  1030.00,  1030.00

reuse/fresh scenario:

Danijel.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,     Auto_Syn ,    500.00  ,  90.00 (   -*          |              ), 480.00,  500.00,  570.00
   2 ,         BM25 ,    620.00  ,  80.00 (        -*     |              ), 600.00,  620.00,  680.00
   3 ,       RANDOM ,    670.00  ,  220.00 (        ---*   |              ), 600.00,  670.00,  820.00
   4 ,        REUSE ,    815.00  ,  110.00 (               |-*            ), 760.00,  820.00,  870.00

K_all3.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,         BM25 ,    680.00  ,  80.00 (     -*        |              ), 660.00,  680.00,  740.00
   1 ,     Auto_Syn ,    710.00  ,  60.00 (      --*      |              ), 680.00,  710.00,  740.00
   2 ,       RANDOM ,    815.00  ,  210.00 (        ------*|              ), 720.00,  830.00,  930.00

Hall.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,         BM25 ,    340.00  ,  50.00 ( -*            |              ), 320.00,  340.00,  370.00
   2 ,       RANDOM ,    355.00  ,  180.00 ( --*           |              ), 320.00,  360.00,  500.00
   2 ,     Auto_Syn ,    380.00  ,  40.00 (   --*         |              ), 360.00,  380.00,  400.00
   2 ,        REUSE ,    385.00  ,  30.00 (    -*         |              ), 370.00,  390.00,  400.00

Wahono.csv

rank ,         name ,    med   ,  iqr
----------------------------------------------------
   1 ,         BM25 ,    630.00  ,  30.00 (    *          |              ), 610.00,  630.00,  640.00
   2 ,        REUSE ,    720.00  ,  60.00 (       *       |              ), 710.00,  720.00,  770.00
   2 ,     Auto_Syn ,    725.00  ,  80.00 (      -*       |              ), 680.00,  730.00,  760.00
   2 ,       RANDOM ,    740.00  ,  360.00 (  ------*      |              ), 550.00,  760.00,  910.00

Stop at 95% relevant found (using estimated number)

Danijel.csv

rank ,         name ,    med   ,  iqr 
----------------------------------------------------
   1 ,         BM25 ,    760.00  ,  50.00 (   *           |              ), 740.00,  760.00,  790.00
   2 ,       RANDOM ,    895.00  ,  330.00 (  ---*         |              ), 650.00,  900.00,  980.00
   2 ,     Auto_Syn ,    920.00  ,  60.00 (     *         |              ), 880.00,  920.00,  940.00
   3 ,   UPDATE_ALL ,    1090.00  ,  10.00 (       *       |              ), 1090.00,  1090.00,  1100.00
   4 ,   UPDATE_POS ,    1325.00  ,  70.00 (          *    |              ), 1280.00,  1330.00,  1350.00
   5 ,        REUSE ,    2745.00  ,  120.00 (               |            -*), 2680.00,  2750.00,  2800.00

K_all3.csv

rank ,         name ,    med   ,  iqr 
----------------------------------------------------
   1 ,         BM25 ,    460.00  ,  20.00 (         --*   |              ), 450.00,  460.00,  470.00
   2 ,       RANDOM ,    480.00  ,  40.00 (           ---*|              ), 460.00,  480.00,  500.00
   3 ,     Auto_Syn ,    520.00  ,  30.00 (               |  ----*       ), 500.00,  520.00,  530.00

Hall.csv

rank ,         name ,    med   ,  iqr 
----------------------------------------------------
   1 ,         BM25 ,    490.00  ,  30.00 (  -*           |              ), 470.00,  490.00,  500.00
   1 ,   UPDATE_POS ,    490.00  ,  50.00 (  -*           |              ), 460.00,  490.00,  510.00
   2 ,       RANDOM ,    545.00  ,  150.00 (  ----*        |              ), 470.00,  550.00,  620.00
   3 ,        REUSE ,    710.00  ,  40.00 (             -*|              ), 690.00,  710.00,  730.00
   4 ,     Auto_Syn ,    750.00  ,  50.00 (               |*             ), 720.00,  750.00,  770.00
   5 ,   UPDATE_ALL ,    940.00  ,  10.00 (               |          *   ), 940.00,  940.00,  950.00

Wahono.csv

rank ,         name ,    med   ,  iqr 
----------------------------------------------------
   1 ,         BM25 ,    1165.00  ,  80.00 (        -*     |              ), 1110.00,  1170.00,  1190.00
   2 ,       RANDOM ,    1280.00  ,  610.00 (     ------*   |              ), 930.00,  1290.00,  1540.00
   2 ,     Auto_Syn ,    1330.00  ,  70.00 (           *   |              ), 1310.00,  1330.00,  1380.00
   3 ,   UPDATE_ALL ,    1510.00  ,  10.00 (              *|              ), 1510.00,  1510.00,  1520.00
   4 ,   UPDATE_POS ,    1750.00  ,  50.00 (               |  *           ), 1740.00,  1750.00,  1790.00
   5 ,        REUSE ,    2350.00  ,  70.00 (               |           -* ), 2320.00,  2350.00,  2390.00

[timm]

From my small-scale experiments, BM25 is really a good strategy for all scenarios. It requires no previous data, little domain knowledge (knowing the keywords to search for).

huh. i wonder how much it changes the prior results?

[azhe825]

@timm Updated some results. BM25 really works well in most cases. How do you interpret the results?

[timm]

is this the reverse of our prior conclusion? that we dont need prior knowledge to do a good job of lit review?

[timm]

and, regardless of that, how does this play with the stopping results?

[azhe825]

is this the reverse of our prior conclusion? that we dont need prior knowledge to do a good job of lit review?

Kind of. UPDATE performs quite good when it can be applied.

• For practical use of the tool, I would suggest BM25:
  • it is reliably better than RANDOM
  • can be applied to any scenario
  • may also be useful when the review goes into a corner and the reviewer has a better view of which direction to go (search when the learner fails to find more relevant studies)
• For paper writing, not sure what to do, mainly because a) BM25 is an existing technique suggested by Cormack. b) BM25 is not consistently the best treatment (there is no such treatment).

regardless of that, how does this play with the stopping results? Still running. Results are expected by Wednesday or Thursday. Meanwhile, I am writing/modifying the introduction of the paper.

[timm]

paper due aug 25 you need to do all that you do, and get it into the paper (which will make it longer) by aug15 (then the editing starts...)

by aug15, if possible, would like to see an intro that summarizes your view oin the new results. prior work on fastread incomplete. did not study the effects of prior knowledge. we find prior knowledge useful, but optional. surprisigly good results can be acheived using simple key word frequencies.

now this is ICSE so the intro is all. short sharp, too the point. tells you what was wrong from before, and what is our BIG and IMPORTANT jump. list of RQs (with results). list on contributions. final para on "structure of this paper". all in 1.5 pages.

please acknowledge this comment

[azhe825]

Acknowledged

[timm]

one thing i would say from the above is that RUSE is quirky. BM25 looks all around more reliable

[azhe825]

Auto_Syn is also not a good option since:

• requires domain knowledge to handcraft a fake ``relevant'' study (harder than come up with some keywords for BM25)
• performance is not stable (sometimes good, sometimes really bad)

[timm]

i look forward to reading your conclusions.... by aug15