search

haitian-sun / GraftNet

BSD 2-Clause "Simplified" License
268 stars 56 forks source link

"Subgraph Retrieval for Baseline Models" in PullNet #27

Closed vongyx closed 2 years ago

vongyx commented 3 years ago

Hello, Dr. Sun, I read your other paper "PullNet", I have some questions about WebQSP for you.

  1. In section 4.4 "Subgraph Retrieval for Baseline Models", you give the answer recall in Table 2, I wonder if the recall is for the original GraftNet model(m=2000).
  2. Whether the gold answers for each question in Pullnet and Graftnet are the same.
  3. I calculate the answer recall using the train sets and the test sets in GraftNet, and I gotta 51.45% answer recall under the only 50% KB setting(m=500). However, Table2 in PullNet gives the 48.5% answer recall (Freebase 50%, m=2000). It's not reasonable that the answer recall when m=500 is higher than when m=2000. I wonder whether I shouldn't have calculated the answer recall using train and test sets. And whether I have some mistakes in understanding this part.
haitian-sun commented 3 years ago

Hi,

Sorry I forgot to reply.

  1. No. The number reported in the paper only use Personalized Pagerank. The retrieval system in GraftNet use some additional heuristics to improve retrieval as well.
  2. I am not sure what do you mean by gold answers here. They are the same dataset so the should have the same answers.
  3. It’s likely due to the difference of the two retrieval systems. (See my reply to your first question).

On May 11, 2021, at 10:49 PM, vongyx @.***> wrote:

Hello, Dr. Sun, I read your other paper "PullNet", I have some questions about WebQSP for you.

In section 4.4 "Subgraph Retrieval for Baseline Models", you give the answer recall in Table 2, I wonder if the recall is for the original GraftNet model(m=2000). Whether the gold answers for each question in Pullnet and Graftnet are the same. I calculate the answer recall using the train sets and the test sets in GraftNet, and I gotta 51.45% answer recall under the only 50% KB setting(m=500). However, Table2 in PullNet gives the 48.5% answer recall (Freebase 50%, m=2000). It's not reasonable that the answer recall when m=500 is higher than when m=2000. I wonder whether I shouldn't have calculated the answer recall using train and test sets. And whether I have some mistakes in understanding this part. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OceanskySun/GraftNet/issues/27, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADE5XLYYBT5LSTNUZOVCLRTTNHUE7ANCNFSM44XUVLPA.

sudarshan77 commented 2 years ago

Hi Dr. Sun, I started to read the PULLNET article after the GRAFTNET. The pointers discussed here, gave some clarity on the idea. Could you please let me know any possibility of releasing the repo (implementation) for the PULLNET in the near future?