Question about negative context representation in context prediction

[lvguofeng1303]

Hi, I read your code and found that there is something i don't understand. In the context prediction module(pretrain_contextpred.py), the negative context representative is obtained by cycle shifting context_rep or substruct_rep.

# cbow
# negative contexts are obtained by shifting the indicies of context embeddings
neg_context_rep = torch.cat([context_rep[cycle_index(len(context_rep), i+1)] for i in range(args.neg_samples)], dim = 0)

# skipgram
#shift indices of substructures to create negative examples
shifted_expanded_substruct_rep = []
for i in range(args.neg_samples):
    shifted_substruct_rep = substruct_rep[cycle_index(len(substruct_rep), i+1)]
    shifted_expanded_substruct_rep.append(torch.cat([shifted_substruct_rep[i].repeat((batch.overlapped_context_size[i],1)) for i in range(len(shifted_substruct_rep))], dim = 0))

My question is why use cycle shifting to get negative rep? Have you tried other methods, such as use random index or use random rep directly? Is it cycle shifting method is better? In skipgram mode, is it more reasonable to randomly sample other nodes as negative rep? Looking forward to your reply, thanks a lot!!!

[weihua916]

Hi! We have only tried the cycle shift setting. Randomly sampling other nodes should also work.