search

snorkel-team / snorkel

A system for quickly generating training data with weak supervision
https://snorkel.org
Apache License 2.0
5.81k stars 857 forks source link

Snorkel MeTaL: Multi-task Basic tutorial example #1552

Closed selectasterisk closed 4 years ago

selectasterisk commented 4 years ago

This is a question regarding this example: https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb

I know that repository has been put on maintenance mode, and that this is where the latest Snorkel development has been happening, but I was curious if the contents of that tutorial is still relevant?

I'm interested in using the framework proposed in the 2019 Paper to tackle a multi-label classification problem, where there might be hierarchical structures in the data. I have several questions:

1) I've read that the implementation in the master branch as it stands now is only for the conditionally independent case, per: https://github.com/snorkel-team/snorkel/issues/1462. Further, the code for the implementation seems to only work for the single-task case. I am curious when there might be a proper implementation of the 2019 paper?

2) Going back to the tutorial example from above; I've noticed that the label matrices have to be [n,m]; where n is the number of datapoints, and m is the number of labeling function outputs. Is there an implementation where the different tasks could vary the number of labeling functions used? (e.g. task 1 has 4 different LF, task 2 uses 3, etc...) Or is there an equivalent algebra work around for this?

Thanks!

ajratner commented 4 years ago

Hi @selectasterisk ,

As you note, that tutorial--which covers the multi-task case--is in the Snorkel MeTaL repo, which is in maintenance mode. Unfortunately there are no active (i.e. near term) plans to implement multi-task functionality in the Snorkel repo (this repo), but obviously we hope to get to it as soon as we can down the road!

Re: your second question, our convention in that implementation was just to include abstains for LFs that don't output labels (or have labels implied by the task structure) for a given task. Another way of thinking of this is that the LFs output vectors which have length equal to the number of tasks for each data point, but which are sparse and may always abstain on certain tasks.

Hope this helps! -Alex