hugochan
commented
6 years ago Follow-up:
I was using the data accompanying this ACL 2014 paper [1]. They provided the dumped results from Freebase Search API. Here is the link to the data: http://cs.jhu.edu/~xuchen/packages/freebase-data.tar
Here is the reply from the authors of the STAGG paper: It’s been a while and I no longer possess all the files when I conducted this research work. However, I believe that we didn’t use Xuchen’s data in the end and called the Freebase API ourselves at that time. As far as I remember, we did very limited preprocessing when using Freebase API. In contrast, Yuchen seemed to have done some important preprocessing work (e.g., named entity and noun phrase detection). This might be the reason for the discrepancy.
[1] Yao, Xuchen, and Benjamin Van Durme. "Information extraction over structured data: Question answering with freebase." Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vol. 1. 2014.
Hi,
Thank you for providing those valuable resources. In the STAGG paper, the authors claimed that S-MART was superior than Freebase Search API for topic entity linking. However, I compared the two tools in terms of recall@k scores on the WebQuestions test set and reached an opposite conclusion. Here are the results I got:
fb search api: r@1: 0.857 r@2: 0.916 r@3: 0.932 r@10: 0.956
smart suggest api: r@1: 0.818 r@2: 0.860 r@3: 0.876 r@10: 0.880
Could you please kindly take a look at this? I was using the dumped results of S-MART in this file webquestions.examples.test.e2e.top10.filter.sid.tsv.
Thank you very much!