Effort for using MAR

[azhe825]

Reviewed 350 docs, 160 of them are "relevant". Cost me 1.5 hours.

Task

Hall.csv without true labels, trying to find studies about "software defect prediction".

Operator

Just me

Result

The output csv file is here

Discussions

• Retrieved more than Hall's relevant.
• Among 350 coded studies, 250 of them are consistent with Hall.
• Among Hall's 106 relevant, 72 of them are corrected coded as relevant in my attempt.
• Human errors/ biases do exist.

CSC 510

Propose:

Whole plan

• I suggest we form groups of 6 students.
• each group, choose their own topics in SE.
• each member in one group review only 200 studies (should be able to finish in about 1 hour)
• 3 parallel reviews per group: A reviews 200, then D starts on top of A, do another 200. Same thing happens to B+E, C+F.
• we will have data for both a) disagreements between reviewers, b) concept drift when different reviewers work on top of others.
• students will then read studies they selected and learn from those.

Reason:

• Students can select their own topics.
• As a student, I would rather work 2-3 hours on some topic I personally interested in than 1 hour on something I care less about.
• Data generated will be extremely useful for me.

[timm]

interesting words. need more context?

[azhe825]

Have updated this issue with more details.

[timm]

Still dont get it. guess we ned to talk