[EXP] Evaluate CDAE, MF-BPR Performance #18

[GangBean]

Background

• Evaluating CDAE, MF-BPR model's performance on Yelp dataset using W&B sweep function

To do

• [x] make sweep config (HP search by bayesian)
  • common configs: learning rate, batch_size, optimizer, weight_decay
  • model-specific configs:
    • CDAE: neg_times, hidden_size, corruption_level, hidden_activation
    • MF-BPR: embed_size
• [x] run sweep agent
• [x] log best HP and best performance (best metric: MAP@10)
• [ ] ablation study
  • CDAE:
    • corruption_level(dropout_rate)
    • hidden_activation: sigmoid vs identity
    • hidden_size
  • MF-BPR:
    • embed_size

※ epoch: 100, early stopping: 5 (MAP@10) fixed

[GangBean]

Result

	Precision@10	Recall@10	MAP@10	NDCG@10	link
CDAE	0.01538	0.0713	0.02222	0.02198	link

• learning rate {1e-2, 1e-3, 1e-4, 1e-5} best: 0.001
• batch size {16, 32, 64, 128, 256} best: 64
• weight decay {1e-1, 1e-3, 1e-5} best: 0.1
• optimizer {adam, adamw} best: adam
• negative sampling size {min: 1, max:100} best: 78
• hidden layer dimensions As CDAE (2015) mentioned, we can see that the performance increases with larger hidden dimensions, but only up to a point. When it becomes large enough, the performance no longer improves and can in fact decrease due to overfitting.
  • 
• corruption level A certain level (around 0.4) of noise can aid in learning, but excessive noise can hinder it.
  • 
• hidden layer activation function The encoder with non-linear function surpasses over that without non-linearity.
  •