search

seomoz / word2gauss

Gaussian word embeddings
MIT License
190 stars 52 forks source link

Entailment experiments not working as original paper #15

Closed shyamupa closed 8 years ago

shyamupa commented 8 years ago

Hi,

I am trying to use the model for doing word entailment (eg. dog --> mammal) as described in the original paper (Section 6.2). I got the data for the experiment from this link.

However, I cannot reproduce the results for the experiment in the paper. Has anyone tried this and succeeded?

matt-peters commented 8 years ago

I haven't tried to reproduce those results. Are you following the same hyperparameter choices/setup? What F1 values are you getting?

shyamupa commented 8 years ago

I ask the author and tried using the following parameters

l2 norm  <100
variance between 0.25 and 1000.0
margin of 5.0
50 dimension
diagonal covariance 
training with IP objective
min-freq for vocab 100

trained on wikipedia. Strangely, I cannot beat the baseline of always predicting True (which gets a F1 of 66%).

matt-peters commented 8 years ago

Agree that's strange. As I said I haven't tried to reproduce those results so unfortunately won't be of much help.