search

snorkel-team / snorkel

A system for quickly generating training data with weak supervision
https://snorkel.org
Apache License 2.0
5.81k stars 857 forks source link

Tie break policy in LabelModel.predict() #1555

Closed durgeshiitj closed 4 years ago

durgeshiitj commented 4 years ago

As per documentation https://snorkel.readthedocs.io/en/v0.9.3/packages/_autosummary/labeling/snorkel.labeling.LabelModel.html#snorkel.labeling.LabelModel.predict while using 'abstain' it should return [-1] as vote in case of conflict.

In our example, we had explicitly mentioned tie-break-policy as 'abstain'. But I got the following predicted labels using LabelModel.predict(tie-break-policy='abstain).

Label-Matrix                                          Predicted-Label
[-1, -1, -1, -1, -1,  5, -1, -1,  9]                       9

It is not returning -1 but predicting class as '9'. So we are wondering what is the meaning of tie-break policy argument that we provide in LabelModel.

brahmaneya commented 4 years ago

Hi @durgeshiitj, the tie-break policy only applies if there is a tie between the probabilities of the most probable labels. In your example, the votes 5 and 9 were by two different LFs. If the LF that voted 9 has a higher accuracy than the other, then the probability of 9 will be higher than that of 5, and there won't be a tie to break.