search

maciejkula / spotlight

Deep recommender models using PyTorch.
MIT License
2.97k stars 421 forks source link

Difference between user and sequence representation #155

Open JoaoLages opened 5 years ago

JoaoLages commented 5 years ago

Can anyone explain me why the user and sequence representation are calculated in this way?? seems like the last state of the LSTM is the sequence representation and the rest is the user representation. I'm not following how this works.

JoaoLages commented 5 years ago

Ok, I'm starting to understand. I missed this pad that you do to your input. You mask the input so that the task is kind of like 'predicting the last input token'. That makes more sense.

However, I am not following why you still mask the input at the end and use the last input representation

JoaoLages commented 5 years ago

While predicting, for me it would make sense if you would use the whole input, non-padded, and the whole output afterwards, not only a portion of it related with the last input token. By that I mean, using the full user_representations variable and not adding any pad.

JoaoLages commented 5 years ago

Ah ok, I think I finally understood. Only user_representations will contain vectors that try to become as clone as the input vector, after passing through the embedding layer. This means that only the last embedding vector of user_representations will actually contain the predictions for the next input token, makes sense.

I will leave this issue open to see if somebody can confirm this.

maciejkula commented 5 years ago

This sounds about right: the representation at step i encodes all user behaviours up to i.

JoaoLages commented 5 years ago

It'd be cool if we could have a custom hidden layer with that behavior to add more non-linearities and transformations to the model

JoaoLages commented 5 years ago

What's the big advantage over training only one time step at a time? By that means, each i would have a single bakprop