search

yanaiela / pararel

MIT License
42 stars 8 forks source link

Which three relations have you filtered and how to get the exactly result presented in table 3? #3

Open Kaiwen-Tang opened 2 years ago

Kaiwen-Tang commented 2 years ago

Hello! Thanks for releasing the code and data. When I tried to reproduce the results, I found there are 39 relations here [https://github.com/yanaiela/pararel/tree/main/data/pattern_ data/graphs_json.] while in the article they should be 38. I wonder if I have miss anything to filter the data and a little confused about which one should be removed.

Meanwhile, I tried to reproduce the results in table 2 and 3. So after filtering, I run run_lm_consistent.py with bert-base-cased as language model. However, the results are slightly different.

Here are the comparison of my result and the original ones represented in your paper.

metrics results original
Consistency 58.72402028656049+-23.441761447607665 58.5+-24.2
Accuracy 43.5833928562685+-26.147602234172046 45.8+-26.1
Unk-Const 48.17776229348506+-22.316262495803235 46.5+-21.7
known-Const 62.54802072504408+-24.32609269135139 63.8+-24.5

Do you have any idea about what I may miss in the procedure and what can I do to get the same result? Thanks for your kindly help.

yanaiela commented 2 years ago

Hi @Kaiwen-Tang, thanks for your interest!

Re the relations, I think in the repo one of the relation has a single pattern, so it's not included. And for the actual relations we used, since some of them are N-M, we didn't use them as part of the analysis, and filtered them out. You can find all the relations we ended up using here.

Re the results comparison, it might be explained using the different patterns. Do you mind trying again using these relations I specified in the wiki, and let me know if the results still differ?