Generative model for multi_class labels

[SoniaBadene]

Description of the Task

I wrote LFs that outputs (0 for Abstain, 1-10 for the classes, 11 for No_relation). Each LF outputs set has only one value between 1 to 10. The last LF output the 11 value if all the LFs output 0 for a candidate (not sure if I should replace all the Abstain values by No_relation or if the generative model will deduce it, because I don't want to have the Abstain class (value 0) as the result ).

I used the GenerativeModel module :

from snorkel.learning import GenerativeModel

gen_model = GenerativeModel(lf_propensity=True)
gen_model.train(L_dev)
dev_marginals = gen_model.marginals(L_dev)

I couldn't use the 46 dependencies that resulted from the following code:

from snorkel.learning.structure import DependencySelector

ds = DependencySelector()
deps = ds.select(L_dev, higher_order=True ,threshold=0.1)
print('#deps: ', len(deps))

Results

I obtain a matrix dev_marginals where for each candidate row, I have a probability of each class (LF). I then select the highest probability and the index to determine which class(LF) has been predicted.

When I count the predicted classes (max of probabilities), I have class 0 Abstain, and not at all class 11 No_relation.

Questions

• Is my way of selecting the highest probability to determine which class was predicted is wrong?
• How to use dependencies for this multi_class problem in the Snorkel generative model? But not sure it's necessary if the LFs don't overlap.
• My LFs output only one class each, or intuitively the generative model models LFs by observing how they overlap and conflict with each other. I don't understand how the generative model could give me those probabilities for each class where my LFs conflics only for 0 (Abstain) and 11 (No_relation)..?
• Is it the small number of candidates where output 11 occurred that makes it unpredictable...(class 11 not the highest probability)? Or the fact that I have 11 classes?

Thank you in advance for your help !

[santosh-b]

The generative model isn't properly deducing your ABSTAIN class because it should have an index of -1, not 0. So stuff like coverage/overlap relating to ABSTAINs would be incorrect

Reference

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.