Open Kaiwen-Tang opened 2 years ago
Hi @Kaiwen-Tang, thanks for your interest!
Re the relations, I think in the repo one of the relation has a single pattern, so it's not included. And for the actual relations we used, since some of them are N-M, we didn't use them as part of the analysis, and filtered them out. You can find all the relations we ended up using here.
Re the results comparison, it might be explained using the different patterns. Do you mind trying again using these relations I specified in the wiki, and let me know if the results still differ?
Hello! Thanks for releasing the code and data. When I tried to reproduce the results, I found there are 39 relations here [https://github.com/yanaiela/pararel/tree/main/data/pattern_ data/graphs_json.] while in the article they should be 38. I wonder if I have miss anything to filter the data and a little confused about which one should be removed.
Meanwhile, I tried to reproduce the results in table 2 and 3. So after filtering, I run
run_lm_consistent.py
with bert-base-cased as language model. However, the results are slightly different.Here are the comparison of my result and the original ones represented in your paper.
Do you have any idea about what I may miss in the procedure and what can I do to get the same result? Thanks for your kindly help.