The Question on Evaluation

[lsq980114]

Hi Sun, In the step Evaluation, we need three file: pred_kb, pred_doc, pred_hybird. But I notice there are only one file setting on .yml likes pred_hybrid in webqsp. So, how could I get the file of pred_kb and pred_doc? Wish for your reply！

[haitian-sun]

The pred files should have the same file format. Can you try replacing the pred_hybrid file with pred_kb and pred_doc?

On Apr 5, 2021, at 9:31 AM, lsq980114 @.***> wrote:

Hi Sun, In the step Evaluation, we need three file: pred_kb, pred_doc, pred_hybird. But I notice there are only one file setting on .yml likes pred_hybrid in webqsp. So, how could I get the file of pred_kb and pred_doc? Wish for your reply！

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OceanskySun/GraftNet/issues/22, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADE5XL6U3GMMSPW7I6NEJPTTHG3RLANCNFSM42MZ7H7A.

[lsq980114]

Hi Sun, Did you means pred_kb was generated with set use_kb "True" and use_doc with False? Wish for your reply！

[haitian-sun]

Yes, I think that’s what it’s designed to be.

On Apr 5, 2021, at 9:55 PM, lsq980114 @.***> wrote:

Hi Sun, Did you means pred_kb was generated with set use_kb "True" and use_doc with False? Wish for your reply！

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OceanskySun/GraftNet/issues/22#issuecomment-813766323, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADE5XL3XB5LBDWFDG2BESWTTHJSZ5ANCNFSM42MZ7H7A.

[haitian-sun]

KB is easy to use if its coverage is complete. The model will tend to use KB and ignore whatever on the text side. That’s why we introduced a fact dropout mechanism to resolve this issue.

In general, as the KB gets less complete, you should see an improvement when adding in text. I am not sure which dataset you use. I recall in our experiment we report in the paper, we start to see an improvement with 50% KB.

Hope this help.

On Apr 23, 2021, at 2:11 AM, vongyx @.***> wrote:

Hi Sun, In the process of recurrent, I found that the HITS and F1 results of only-KB were higher than those of KB + Text under full KB and 50% KB Settings. Why is that? Wish for your reply！

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OceanskySun/GraftNet/issues/22#issuecomment-825414792, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADE5XLYCV4V6YP5F7QE4D23TKEFQJANCNFSM42MZ7H7A.

[coco07]

Hi Sun, Did you means pred_kb was generated with set use_kb "True" and use_doc with False? Wish for your reply！

Hi,

Even after changing the config accordingly, I am not able to generate pred_kb? If you have found, can you please tell me from where it will get generated?

Edit: Finally, I found out how to do it

wikimovie: python3 script.py wikimovie ./model/wikimovie/pred_doc ./model/wikimovie/pred_doc ./model/wikimovie/pred_doc

webqsp: python3 script.py webqsp ./model/webqsp/pred_hybrid ./model/webqsp/pred_hybrid ./model/webqsp/pred_hybrid

and as mentioned above, if we keep use_doc as false in webqsp.yaml, it will generate pred_kb. But because we have kept the name of file as pred_hybrid in the config file, we are not able to see use_kb file (instead we see pred_hybrid only)