Parameters for reproducing results from paper

[bkj]

Can you provide the parameters for reproducing the results from the paper on FB15k and FB15K-237? I ran the command from the README:

 CUDA_VISIBLE_DEVICES=0 python main.py --dataset FB15k-237 --num_iterations 500 --batch_size 128
                                       --lr 0.0005 --dr 1.0 --edim 200 --rdim 200 --input_dropout 0.3 
                                       --hidden_dropout1 0.4 --hidden_dropout2 0.5 --label_smoothing 0.1

which gave final performance of

Number of data points: 35070
Hits @10: 0.4009124607927003
Hits @3: 0.2555460507556316
Hits @1: 0.1760193897918449
Mean rank: 291.46401482748786
Mean reciprocal rank: 0.24741750020439274
Test:
Number of data points: 40932
Hits @10: 0.3974396560148539
Hits @3: 0.2546662757744552
Hits @1: 0.17094205022964917
Mean rank: 304.61949086289457
Mean reciprocal rank: 0.24344486414937788

Any ideas?

UPDATE: I noticed in the paper that you mention the best learning rate for FB15k-237 is 0.005 instead of 0.0005 and best the learning rate decay is 0.995 instead of 1.0 -- might that be the issue?

[ibalazevic]

The command you ran is the one that should reproduce the results from the paper. Actually, in the paper we say (0.0005, 1.0) are the best learning and decay rate for FB15k-237 and (0.005, 0.995) are the best ones for WN18. Either way, from my experience, whichever learning rate and decay rate you choose, it shouldn't influence the results that much (definitely not by almost 15% on hits@10).

I just reran the command from README and got these numbers:

Test:
Number of data points: 40932
Hits @10: 0.5443513143750611
Hits @3: 0.39400938141307534
Hits @1: 0.26598867389817256
Mean rank: 158.28904035962083
Mean reciprocal rank: 0.3578093749942233

I'm not really sure why your numbers are so different, are you maybe using a different version of PyTorch?

[bkj]

I was using torch==1.0.0 -- I'll try again w/ torch==0.4.0 and let you know.

(And noted on the parameters -- I misread the paper. My mistake!)

[bkj]

I figured out what was going on -- I had to make some small changes to get torch==1.0.0 to stop complaining, and I made a small mistake. Thanks for your help!

[bkj]

Do you happen to have the exact parameters you used for the other experiments?

[ibalazevic]

I've just updated the README with hyperparameter combinations for all the datasets.

[bkj]

Thanks!

On Fri, Feb 15, 2019 at 11:15 AM Ivana Balazevic notifications@github.com wrote:

I've just updated the README with hyperparameter combinations for all the models.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ibalazevic/TuckER/issues/2#issuecomment-464107508, or mute the thread https://github.com/notifications/unsubscribe-auth/AFzgfftRWo5YDrk9QS2uoTv_mBR9RFxgks5vNt01gaJpZM4a82FC .

[l6270789]

I used torch==1.0.0 and cannot reproduce the results from the paper. What changes do you make from torch==0.4.0 to torch==1.0.0? Thanks for your help!