webqsp dataset: full 0.439317 accuracy, half 0.346397

[lihuiliullh]

May I know how do you run your model in webqsp dataset? I use the following command to run the model on webqsp half

python3 main.py --mode train --relation_dim 200 --do_batch_norm 1 \ --gpu 2 --freeze 1 --batch_size 16 --validate_every 10 --hops webqsp_half --lr 0.00002 --entdrop 0.0 --reldrop 0.0 --scoredrop 0.0 \ --decay 1.0 --model ComplEx --patience 20 --ls 0.05 --l3_reg 0.001 --nb_epochs 200 --outfile half_fbwq

and the following command to run the model on webqsp full python3 main.py --mode train --relation_dim 128--do_batch_norm 1 \ --gpu 2 --freeze 1 --batch_size 16 --validate_every 10 --hops webqsp_half --lr 0.00002 --entdrop 0.0 --reldrop 0.0 --scoredrop 0.0 \ --decay 1.0 --model ComplEx --patience 20 --ls 0.05 --l3_reg 0.001 --nb_epochs 200 --outfile half_fbwq

But the accuracies are full 0.439317 accuracy, half 0.346397.

[apoorvumang]

are you using the pre-trained complex embeddings? @lihuiliullh

[lihuiliullh]

yes, I think so. Because I downloaded the pretrained_models.zip, and used the above command line. @apoorvumang

[lihuiliullh]

wait for a while, let me get back to you later. Because I found that the "nb_epochs" I used is 20 instead of 200.

[apoorvumang]

did you get a chance to run with 200 epochs? @lihuiliullh

[lihuiliullh]

yes, I ran 200 epochs. The accuracy for full is 0.56, for half is 0.47. @apoorvumang How do you get the accuracy in the paper?

[apoorvumang]

do these numbers include relation matching, or just the output of training script? @lihuiliullh

[lihuiliullh]

How to do relation matching? I just follow the instruction on the github, and ran the command. @apoorvumang

[apoorvumang]

please see this https://github.com/malllabiisc/EmbedKGQA/blob/master/KGQA/RoBERTa/README.md

[lihuiliullh]

Do you have half version dataset like webqsp_scores_full_kg_fixed.pkl? If I want to run the half datset, what should I do? @apoorvumang

[lihuiliullh]

Also, may I know how do you get the candidates in webqsp_scores_full_kg_fixed.pkl? Are they the top-200 prediction of the very basic model without relation matching and 2-hop filter?

[apoorvumang]

1. I do not yet have saved scores for half-kg model output. Will update you once I get it. You can however get scores from your own model through the instructions outlined in the above mentioned readme (you will have to uncomment a line in code to save file)

2. The given scores file contain top-200 prediction without relation matching, without 2-hop filter

[apoorvumang]

closing since remaining points have been raised in another issue