Plot the estimated curve

[azhe825]

Why

• provide an idea of how many 'relevant' are there in the unlabeled candidate set #57
• estimate the precision (efficiency) of future review-- how many relevant can be found if we review N documents
• so that user can decide when to stop

Method

57

Estimate at "relevant found" = 30 (when aggressive undersampling is first applied)

Hall

Wahono

Danijel

Conclusion

• not great at long term estimation
• quite good at short term estimation, especially next round
• from #57, good at estimate total 'relevant' numbers

Further Discussion

• long term estimation

  • why it is not good: classifier get improved every round
  • is it necessary?
  • how to make it better? simulate the process perhaps

• what does user really want?

  • how we present the estimation?

Compensate for the under-estimation

Hall

Wahono

Danijel

[timm]

shit. you've KILLED the estimation problem

ICSE paper? when to stop reading?

what is the method?

[timm]

now if you onyl had some way to determine how error tolerant is your rig.

even if we dont have people, can we say that errors less than X% do not threaten our conclusions?

[azhe825]

For curve estimation:

The method is semi-supervised learning, details in #57 Doing more experiments on CLEF data sets to see if the estimation is good always.

For Error tolerant:

Have some initial idea of simulating human errors:

• for each unlabeled data point, draw a circle around it
• there is a chance of human misclassisify the target data point to the class of one of the point lands inside the circle (BENEFIT: so that dog might be misclassified as cat but never as elephant)
• can change the size of the circle and the chance of misclassification to see what our tolerance is