search

mana-ysh / knowledge-graph-embeddings

Implementations of Embedding-based methods for Knowledge Base Completion tasks
Apache License 2.0
257 stars 63 forks source link

Inconsistent results #13

Closed aishahasmoh closed 5 years ago

aishahasmoh commented 5 years ago

I have been trying to replicate the results you are reporting for ComplEx on WN18 but I can't get the same results at all using the exact same hyper-parameters you are using. The command I am running for training is:

python3.6 train.py --ent  wn18_entities --rel wn18_relations --train wordnet-mlj12-train.txt --valid wordnet-mlj12-valid.txt --method complex --epoch 500 --batch 128 --lr 0.05 --dim 200 --negative 5 --opt adagrad --l2_reg 0.001 --gradclip 5 --metric hits --nbest 1 --filtered  --graphall wordnet-mlj12-train.txt --log log/

Notice that I am using the training data as input to --graphall argument. Can you please confirm if I should use that.

For testing I am using the following command:

 python3.6 test.py --ent wn18k_entities --rel wn18k_relations --data wordnet-mlj12-valid.txt --filtered --graphall wordnet-mlj12-train.txt  --method complex --model log/ComplEx.best

and the results I am getting are:

Hits@1              : 0.4838
Hits@1(filter)      : 0.8097
Hits@10             : 0.8037
Hits@10(filter)     : 0.927
Hits@3              : 0.6686
Hits@3(filter)      : 0.8897
MRR                 : 0.5954195358197126
MRR(filter)         : 0.8545953294974848

Can you please explain what could be the reason this is happening?

aishahasmoh commented 5 years ago

I realized that I have to use a concatenation of all the training, validation and testing data for the --graphall argument to get the results reported here.