First exp on Syn_Error

[azhe825]

Syn_Error:

• lda to transform each doc into a vector of M=100
• For each doc i:
  • find neighbors: find all doc j that cosine_dist(vec(i), vec(j)) < thres=0.1 (including i itself)
  • when labeling doc i, the human label is chosen randomly from the set of doc j
  • which means
    • if doc i=1000 has 5 neighbors {23, 101, 1000, 3122, 4111}
    • 23, 101, 4111 have true label 'no' while 1000, 3122 have true label 'yes'
    • then doc i=1000 has a probability of 0.4 to be labeled as 'yes' and a probability of 0.6 to be labeled as 'no'

Syn_Error Result:

Hall data set

• 16 'no' have a chance to be labeled as 'yes'
• 22 'yes' have a chance to be labeled as 'no'

Experiment on Syn_Error data:

Stop rule:

• Find more than 95% 'yes' docs, comparing to the estimated number

Result:

• 9 'yes' have been wrongly labeled as 'no'
• 6 'yes' have not been reviewed
• 8 'no' have been wrongly labeled as 'yes'

[timm]

v.interesting

what i'd ike to see is the "breakdown point"; i.e. is there a level of error after which the results decay dramatically

[azhe825]

Should be able to do this by controlling the thres = 0.1. Making thres larger should increase the chance of one doc being wrongly labeled.

Currently running exps to test the effect of different thres