snorkel-team / snorkel

A system for quickly generating training data with weak supervision
https://snorkel.org
Apache License 2.0
5.79k stars 859 forks source link

Create Scoped Categoricals #649

Closed ajratner closed 7 years ago

ajratner commented 7 years ago

Create mapping in Snorkel front-end so that each Candidate (variable) can have its own support; should then require minimal changes to compiler (just changing cardinality) and none to NS!

Example: suppose that we have a closed-world entity-linking task, e.g. mapping disease mentions to MESH IDs. If we have a mention like "headache", we don't want to consider all possible IDs (something like 40,000 I think?) during learning / inference; instead we probably have a small list of IDs to consider (e.g. the set of all labels given by an LF).

@jason-fries for EL stuff!

Working on branch scoped_categoricals; to-dos:

ajratner commented 7 years ago

Notes:

ajratner commented 7 years ago

Want to write one more test, then done for now