Unable to replicate oracle accuracies for evolved type systems

openai / deeptype

Code for the paper "DeepType: Multilingual Entity Linking by Neural Type System Evolution"

https://arxiv.org/abs/1802.01021

Other

647 stars 147 forks source link

Unable to replicate oracle accuracies for evolved type systems #27

Open MurtyShikhar opened 6 years ago

MurtyShikhar commented 6 years ago

Hi, I'm not able to replicate the results of Table 1(a). The oracle accuracy of the type system evolved via CEM that I obtain is ~92.5% whereas the human designed type system obtains an oracle accuracy of ~98%.

Could you please share the JSONs for the evolved type systems if possible? Congrats, btw on the amazing work and the really well written code-base.

-Shikhar