search

openai / deeptype

Code for the paper "DeepType: Multilingual Entity Linking by Neural Type System Evolution"
https://arxiv.org/abs/1802.01021
Other
647 stars 147 forks source link

Evaluate Learnability does not give output graph as per the jupyter notebook LearnabilityStudy.ipynb #42

Closed heisenbugfix closed 5 years ago

heisenbugfix commented 5 years ago

I ran the evaluate_learnability.py code by creating sample_data.tsv (containing 1000 samples as per the config file). I got following warning. /usr/local/lib/python3.5/dist-packages/sklearn/metrics/ranking.py:571: UndefinedMetricWarning: No positive samples in y_true, true positive value should be meaningless UndefinedMetricWarning

  1. What can be done for the above? Is the sampled articles or their number(1000 seems less) a problem here?
  2. I ran the LearnabilityStudy.ipynb file and the figures I obtain are not as per what is shown in the paper. I have attached the figures. Please help me with same. auc-hist.pdf auc-dev-frequency.pdf auc-std.pdf