search

zyang1580 / CoLLM

The implementation for the work "CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation".
BSD 3-Clause "New" or "Revised" License
45 stars 4 forks source link

Hyperparameters to reproduce the result of collab. model on amazon book dataset? #5

Open YifanHao opened 5 months ago

YifanHao commented 5 months ago

Hi! I've read the paper and found it very interesting so I'm trying to reproduce it. However, I'm kind of stucked in the very first step to train the base collab. model. I used baseline_train_sasrec_amazon.py and baseline_train_mf_ood_amazon.py to train SASRec and MF model seperately, with the hyperparameters in the scripts unchanged. (except batch_size: 10240 -> 1024 in baseline_train_sasrec_amazon.py, I thought it might be a typo since the value is always 1024 in baseline_train_sasrec.py, baseline_train_mf_ood.py and baseline_train_mf_ood_amazon.py) But my result is much lower than those reported in the paper. So I wonder if it's the hyperparameters in the scripts are just for experiments and not the optimal values? Or something I didn't notice may cause the gap?

The training logs produced by my run are as follows:

# SASRec
train_config: {'lr': 0.01, 'wd': 0.0001, 'embedding_size': 64, 'epoch': 5000, 'eval_epoch': 1, 'patience': 50, 'batch_size': 1024, 'maxlen': 20} 
best result: {'valid_auc': 0.6550170900138883, 'valid_uauc': 0, 'test_auc': 0.6478201601802253, 'test_uauc': 0, 'epoch': 64}
# MF
train_config: {'lr': 0.001, 'wd': 0.0001, 'embedding_size': 256, 'epoch': 5000, 'eval_epoch': 1, 'patience': 50, 'batch_size': 1024} 
best result: {'valid_auc': 0.5837625758627063, 'valid_uauc': 0.5173472295556202, 'test_auc': 0.5749810031561386, 'test_uauc': 0.5242115883441811, 'epoch': 10}

Thanks for you advices!

Update: It seems setting batch_size to 10240 make sense for SASRec on Amazon dataset. My result is close to the value reported in the paper after doing so. For MF model, weight_decay seems to be the key parameter, the performance boosts after I set it to 1e-5.

zyang1580 commented 4 months ago

Sorry for the late reply.

Within the training file for the baseline, there are three sections commencing with if __name__=='__main__':, with the first two sections commented out. The initial section (here), which is commented out, contains the hyperparameter tuning code, offering default values for parameters such as batch size.