greenelab / snorkeling

Extracting biomedical relationships from literature with Snorkel 🏊
Other
58 stars 17 forks source link

Generative Model Benchmarking and bug fixes. #66

Closed danich1 closed 5 years ago

danich1 commented 5 years ago

This is PR is part of the refactoring process. There are minor script changes to fix bugs or add needed functions. The bigger change is the notebook that is being uploaded. It contains the results for bench-marking the generative model. Finally figured out the problem, which was that there wasn't enough regularization and the model weights were doing whatever they wanted. Now that has been solved, I have achieved aurocs close to 70%.

dhimmel commented 5 years ago

In the following plot, is the x-axis the regularization parameter?

auc

At low values, the performance is erratic. Erratic is bad. Why is the performance erratic? Personally 0.4 looks best to me, but I'd like to understand how the erratic performance is sometimes giving such high AUROC.

Let's chat.