datawhalechina / torch-rechub

A Lighting Pytorch Framework for Recommendation Models, Easy-to-use and Easy-to-extend.
MIT License
400 stars 74 forks source link

Bug fix: the forward method from SINE model should return shape (batch_size, n_pos + n_neg) #32

Closed bokang-ugent closed 2 years ago

bokang-ugent commented 2 years ago

Three changes are made in this commit:

  1. SINE model forward function. The returned value of the forward function in SINE model used to sum over dim=1, see this line, returning a tensor with shape (batch_size, embedding_dim). However, this is incorrect, it should be sum along the embedding dimension in order to yield a tensor with shape (batch_size, n_pos+n_neg). This is fixed in this commit.
  2. Train/test user historical sequence generating scheme. The padding and truncating now are changed to "pre", yielding a more appropriate historical sequence.
  3. Update default hyperparameters to the ones used in the original SINE implementation. Before fixing the forward function bug (point 1), the training was not stable and model can easily get overfitted. With strong regularization hyperparamter on Movielens 1M benchmark gives test scores: {'NDCG': ['NDCG@100: 0.0012'], 'MRR': ['MRR@100: 0.0064'], 'Recall': ['Recall@100: 0.1121'], 'Hit': ['Hit@100: 0.1121'], 'Precision': ['Precision@100: 0.0011']}

After the bug was fixed, the training behavior is stabilized, thus the hyperparameters are adjust to allow more epochs, larger learning rate, and less regularization. Movielens 1m test scores are now: {'NDCG': ['NDCG@100: 0.0022'], 'MRR': ['MRR@100: 0.0137'], 'Recall': ['Recall@100: 0.2026'], 'Hit': ['Hit@100: 0.2026'], 'Precision': ['Precision@100: 0.002']}

After changing the historical sequence generating scheme from "post" to "pre", the test scores are further improved: {'NDCG': ['NDCG@100: 0.0052'], 'MRR': ['MRR@100: 0.0382'], 'Recall': ['Recall@100: 0.4298'], 'Hit': ['Hit@100: 0.4298'], 'Precision': ['Precision@100: 0.0043']}