search

RUCAIBox / RecBole-GNN

Efficient and extensible GNNs enhanced recommender library based on RecBole.
MIT License
167 stars 37 forks source link

[🐛BUG] SGL emb_loss should not be averaged. #67

Closed downeykking closed 1 year ago

downeykking commented 1 year ago

In RecBole-GNN SGL, the emb_loss has been averaged: https://github.com/RUCAIBox/RecBole-GNN/blob/77d76b9e19436220b32fe61af95ffd95f17c1db7/recbole_gnn/model/general_recommender/sgl.py#L61

https://github.com/RUCAIBox/RecBole/blob/169412a91f5e504e620d6f708a62f5a7da42580a/recbole/model/loss.py#LL80C23-L80C23

emb_loss /= embeddings[-1].shape[0]

And In paper author's code: the emb_loss hasn't been averaged: https://github.com/wujcan/SGL-Torch/blob/c56328817daa6ebd7669cb9c0429a1f0f3dad5b9/util/pytorch/loss.py#L103

for w in weights:
    loss += torch.sum(torch.pow(w, 2))

Although it doesn't affect the model performance.

hyp1231 commented 1 year ago

Thanks for your clarification! sum or mean could correspond to different learning rates. Glad to hear that in practice they have similar performance.