search

snorkel-team / snorkel

A system for quickly generating training data with weak supervision
https://snorkel.org
Apache License 2.0
5.79k stars 859 forks source link

Details about the generative model (Just a question) #736

Closed shun-zheng closed 6 years ago

shun-zheng commented 7 years ago

Hi, I'm very interested in this awesome system for information extraction. But when I tried to dive into details about the data programming paradigm, I find few materials that describe the detailed learning or inference procedure of the factor graph. In snorkel, this generative modeling is done by calling another system, numbskull.

I have checked the pseudo code in the paper that performs a gradient step like sampling plus sgd. I feel that it's a little bit similar to contrastive divergence for the products of experts model, but I'm not sure. I'm not very familiar with PGM, do you have any reference about this?

Thanks!

ajratner commented 7 years ago

Hi @dolphin-zs ,

Thanks for the interest, and sorry for the delayed response!

Numbskull is used as the learning & inference engine in Snorkel to do Gibbs sampling and stochastic gradient descent for learning the generative label model in snorkel (the GenerativeModel class; represented as a factor graph). We indeed interleave Gibbs sampling and SGD steps, which is very similar to contrastive divergence; you can check out one of our latest papers for slightly more detail here: https://github.com/HazyResearch/numbskull.

Overall, a great reference for PGMs more broadly is the Koller and Friedman textbook. Hope this helps!

-Alex

shun-zheng commented 7 years ago

Hi @ajratner ,

Thanks for your reply! Indeed I plan to learn more about PGM by the textbook of Koller and Friedman. You seems to mention that numbskull has relevant papers? But I haven't found them on the github page of numbskull, haven't them been published yet?

Thanks! -Shun

ajratner commented 7 years ago

No papers for Numbskull yet but we'll post everything to the snorkel.stanford.edu page whenever stuff comes out!

shun-zheng commented 7 years ago

Thanks very much, and looking forward to your new breakthroughs!

shun-zheng commented 7 years ago

Hi @ajratner ,

I find that this paper of your group has properly addressed my questions, which states details about the learning process. I think others interested in details about the denoising generative model may find it to be useful too.

Thanks for your kind reply. -Shun

ajratner commented 7 years ago

Great to hear!

ajratner commented 6 years ago

Closing but will be accessible via the Q&A link!