search

shijx12 / TransferNet

Pytorch implementation of EMNLP 2021 paper "TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph "
63 stars 18 forks source link

WebquestionSP result lower than 0.714 #6

Open xianghuisun opened 2 years ago

xianghuisun commented 2 years ago

Screenshot from 2021-12-02 20-36-21

This is the experimental result on WebQSP(No code changes) My question is that you said the WebQSP had 572 predicates in the paper. And your code is Screenshot from 2021-12-02 20-44-37

but the fbwq_full KG I got from https://drive.google.com/drive/folders/1RlqGBMo45lTmWz9MUPTq-0KcjSd3ujxc is Screenshot from 2021-12-02 20-46-17 : so should I experiment with relation_ids.del instead of relations.dict

Originally posted by @xianghuisun in https://github.com/shijx12/TransferNet/issues/3#issuecomment-984599269

shijx12 commented 2 years ago

Hi, there should be 1144 relations actually. The number in the paper is wrong.

For the inconsistent results on WebQSP, we think it is due to the different versions of Transformers package and its BERT weights. We are trying to recover our previous experimental environment. We conducted the experiments about one-year ago and our machine has been upgraded, so it may take some time.

We have provided the checkpoints in https://cloud.tsinghua.edu.cn/f/786b9853c1d840578025/?dl=1. You can use it for inference.

Huiopfsdfsdf commented 1 year ago

嗨，实际上应该有 1144 个关系。论文中的数字是错误的。

对于WebQSP上不一致的结果，我们认为这是由于不同版本的变压器包及其BERT权重造成的。我们正在努力恢复以前的实验环境。我们大约在一年前进行了实验，我们的机器已经升级，所以可能需要一些时间。

我们在 https://cloud.tsinghua.edu.cn/f/786b9853c1d840578025/?dl=1 提供了检查站。您可以将其用于推理。

Hello brother, is there a log of cwq? Thank you very much