Unable to reproduce results on WN18RR

[apoorvumang]

Hi Ivana

I am trying to get entity embeddings for a downstream application. For WN18RR dataset I was unable to reproduce the reported results of TuckER. I used the hyperparameters given in the README of this repo. Following is the command I used:

 CUDA_VISIBLE_DEVICES=3 python main.py --dataset WN18RR --num_iterations 500 --batch_size 128 \
                                       --lr 0.01 --dr 1.0 --edim 200 --rdim 30 --input_dropout 0.2 \
                                       --hidden_dropout1 0.2 --hidden_dropout2 0.3 --label_smoothing 0.1

And the results are:

495
12.792492151260376
0.00035594542557143403
Validation:
Number of data points: 6068
Hits @10: 0.5121951219512195
Hits @3: 0.4728081740276862
Hits @1: 0.43638760711931446
Mean rank: 6254.662491760053
Mean reciprocal rank: 0.4624483298017613
Test:
Number of data points: 6268
Hits @10: 0.5140395660497766
Hits @3: 0.4738353541799617
Hits @1: 0.43123803446075304
Mean rank: 6595.924856413529
Mean reciprocal rank: 0.45961590280892123
5.328977823257446

Should I increase the number of epochs or am I missing something?

Thanks

[ibalazevic]

Hi Apoorv, yes, you're right, I realized that recently too. The reason is that I initially ran the model with an object entity-specific bias added at the end (before passing the result to the logistic sigmoid). I removed the bias at some point, tested it on FB15k-237 and the results didn't change, however it seems like the absence of the bias has a slight impact on the WN18RR results. I suggest you try either a) adding the bias or b) tuning the hyper-parameters, since it might be that a different hyper-parameter setting will get you the reported results. In any case, please let me know what the outcome is.

[apoorvumang]

I've added a new parameter

self.bias = torch.nn.Parameter(torch.tensor(np.random.uniform(-1, 1, len(d.entities)), 
                                    dtype=torch.float, device="cuda", requires_grad=True))

and did this before sigmoid

x = x + self.bias
pred = torch.sigmoid(x)

Is this correct? I will let you know the results I get

[ibalazevic]

Yes, exactly, except that I initialized it at 0.

[apoorvumang]

Cool, I'll run this and see, thanks

[apoorvumang]

Hey, I was unable to reproduce the results with that change as well. I tried 0 initialization for the bias too. I also got a lot of difference between my multiple runs - seems there is very high variance when I try to run with the same hyperparameters and same initialization(by a lot I mean +-0.2%, ofc thats not much but here its so competitive)

[ibalazevic]

Hi, I've just ran it with a lower learning rate 0.005 and the following code:

x += self.b.expand_as(x)
pred = torch.sigmoid(x)

and was able to reproduce the results.

[apoorvumang]

Thanks, will try it

On Fri, Oct 25, 2019 at 5:09 PM Ivana Balazevic notifications@github.com wrote:

Hi, I've just ran it with a lower learning rate 0.005 and the following code:

x += self.b.expand_as(x) pred = torch.sigmoid(x)

and was able to reproduce the results.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ibalazevic/TuckER/issues/15?email_source=notifications&email_token=AAO6ADZVVFQZRCHK3FAQMPLQQLLGFA5CNFSM4I675DFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECIC5SI#issuecomment-546320073, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO6AD6OKOTCWMJLE4QD4VLQQLLGFANCNFSM4I675DFA .

[ibalazevic]

Just an update: with learning rate 0.003 and running TuckER for 1000 iterations, I actually managed to improve upon the WN18RR results reported in the paper:

Test:
Number of data points: 6268
Hits @10: 0.5295149968091896
Hits @3: 0.4856413529036375
Hits @1: 0.4467134652201659
Mean rank: 6015.324505424378
Mean reciprocal rank: 0.4744804782257188

[luffycodes]

hey, i was wondering if adding the bias at the end (x = x + self.bias) will still count as a valid tucker decomposition?