TA-x model reproduction.

[mer1988]

Hi Alberto, I have been trying for a while to replicate the results from the time aware link prediction paper using the published datasets using the re-implementation of your models from this repo, https://github.com/INK-USC/RE-Net I have carefully looked into the code it looks reasonable to your implementation. The only change I have made from this code and my version is that I change the way they calculate filter hits@10 -> they suppress every object from the ranked list that is observed in training/validation regardless of the time when it occurred while I only suppress those observed in the same timestamp, otherwise I keep it.

The best results I have gotten so far are

  | MRR . | H1 | H10 TA-DistMult | 0.351600377 | 0.2187883521 | 0.628807319 TA-TransE | 0.2173162281 | 0 | 0.6098404552

Do you have any thoughts?? or maybe an accessible implementation of your model?

Thanks!

[mniepert]

Dear Miguel,

Could you provide more context? What dataset is this? One difference in the code that I noticed is that they only use one negative sample (and binary crossentropy) and we use 500 negative samples. This has been shown to make a huge difference for scoring functions such as DistMult.

[mer1988]

Thanks Mathias, I will take a look a close look at neg sampling. The results I show are for the icews14 dataset published in this repo.

Miguel R.

[AGDuran]

Hi miguel, did you apply that filter setting only to compute hits@10 or also for all other metrics?

And yes, the loss is super important to get good performance.

Alberto

El mié., 11 sept. 2019 16:52, Miguel Rodríguez notifications@github.com escribió:

Thanks Mathias, I will take a look a close look at neg sampling. The results I show are for the icews14 dataset published in this repo.

Miguel R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nle-ml/mmkb/issues/3?email_source=notifications&email_token=ACX4SDY4GZD55VZFT3YUQ6TQJEA3XA5CNFSM4IVN5M7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6OYPUY#issuecomment-530417619, or mute the thread https://github.com/notifications/unsubscribe-auth/ACX4SD44SWL7QZV4MTBAWK3QJEA3XANCNFSM4IVN5M7A .

[mer1988]

Hi Alberto, I apply the filter for all metrics. Thanks for the neg sampling suggestion, that I had totally missed.

[mer1988]

Hi Alberto, is there any possibility to use your implementation of the TA-xx models for research purposes?

[AGDuran]

Hello Miguel,

I have a Keras implementation that is super dirty. I dont want to release that implementation without cleaning it up before, but I am busy at the moment with other projects.

I can copy and paste the lines of the model though. It is a very simple code, so I hope you can understand it.

Alberto

TA-transE model

act_fn = None hidden_state = True drop = 0.4 embedding_dim = 100 n_epochs = 500 num_negative = 250

Inputs

e1 = Input(shape=(num_negative + 1,), name="e1") e2 = Input(shape=(num_negative + 1,), name="e2") relation = Input(shape=(9,), name="rel") # Number of tokens in the relation sequence (depends on the dataset)

Entity embeddings

ent_embedding = Embedding(numEnt, embedding_dim, name='ent_embeddings') e1_emb = Dropout(drop)(ent_embedding(e1)) e2_emb = Dropout(drop)(ent_embedding(e2))

Time-enriched relation embedding

rel_embedding = Embedding(numRel + numYear + numMonth + numDay, embedding_dim) rel_emb = Dropout(drop)(rel_embedding(relation)) date_embedding = LSTM(embedding_dim, activation = act_fn, return_state = True) output_state, state_h, state_c = date_embedding(rel_emb) i f cell_state: state = state_c else: state = state_h

Scoring function

e2_emb_minus = Lambda(lambda x: -1 * x)(e2_emb) joint_emb = Add(name='joint_emb')([e1_emb, state, e2_emb_minus])

score1 = Multiply()([joint_emb, joint_emb]) score = Lambda(lambda x: -K.sum(x, axis=2), output_shape=(num_negative + 1,))(score1)

Loss

score = Activation('softmax')(Reshape((num_negative + 1,), name='soft_out')(score)) adam = Adam()

Training

model = Model(input=[e1, e2, relation], output=[score]) model.compile(loss='categorical_crossentropy', optimizer=adam)

El mié., 25 sept. 2019 a las 20:08, Miguel Rodríguez (< notifications@github.com>) escribió:

Hi Alberto, is there any possibility to use your implementation of the TA-xx models for research purposes?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nle-ml/mmkb/issues/3?email_source=notifications&email_token=ACX4SD4KOOTYGL5OP6T3TXLQLOSK3A5CNFSM4IVN5M7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7S2JKA#issuecomment-535143592, or mute the thread https://github.com/notifications/unsubscribe-auth/ACX4SDYMRE7FPEPPIHV3HE3QLOSK3ANCNFSM4IVN5M7A .