search

RUCAIBox / RecBole

A unified, comprehensive and efficient recommendation library
https://recbole.io/
MIT License
3.44k stars 615 forks source link

[🐛BUG] 关于SINE在ml-1m数据集上的结果异常问题。 #1172

Open Sword-CS opened 2 years ago

Sword-CS commented 2 years ago

描述这个 bug 将SINE运行在ml-1m数据集上时，NDCG@10和HIT@10运行结果（NDCG@10=3.88，HIR@10=8.41）远低于原始论文中的结果（NDCG@10=7.06，HIR@10=16.34），认为结果异常。关键参数采用和原始论文一样的配置，即序列长度=20，embedding_size=128，λ=0.5，interest_size=4，concepts=50。请问导致这种问题的可能原因是什么？

以下是在ml-1m数据集上运行SINE时的相关配置及输出信息： 03 Mar 01:41 INFO General Hyper Parameters: gpu_id=0 use_gpu=True seed=2020 state=INFO reproducibility=False data_path=dataset/ml-1m

Training Hyper Parameters: checkpoint_dir=saved epochs=500 train_batch_size=1024 learner=adam learning_rate=0.001 training_neg_sample_num=1 eval_step=1 stopping_step=5

Evaluation Hyper Parameters: eval_setting=TO_LS,full group_by_user=True leave_one_num=2 real_time_process=True metrics=['Recall', 'MRR', 'NDCG', 'Hit', 'Precision'] topk=[10] valid_metric=MRR@10 eval_batch_size=2048

Dataset Hyper Parameters: field_separator=
seq_separator= USER_ID_FIELD=user_id ITEM_ID_FIELD=item_id RATING_FIELD=rating TIME_FIELD=timestamp seq_len=None LABEL_FIELD=label threshold=None NEGPREFIX=neg load_col={'inter': ['user_id', 'item_id', 'rating', 'timestamp']} unload_col=None additional_feat_suffix=None max_user_inter_num=None min_user_inter_num=None max_item_inter_num=None min_item_inter_num=None lowest_val={'rating': 3} highest_val=None equal_val=None not_equal_val=None drop_filter_field=True fields_in_same_space=None fill_nan=True preload_weight=None drop_preload_weight=True normalize_field=None normalize_all=True ITEM_LIST_LENGTH_FIELD=item_length LIST_SUFFIX=_list MAX_ITEM_LIST_LENGTH=20 POSITION_FIELD=position_id HEAD_ENTITY_ID_FIELD=head_id TAIL_ENTITY_ID_FIELD=tail_id RELATION_ID_FIELD=relation_id ENTITY_ID_FIELD=entity_id

03 Mar 01:41 INFO ml-1m The number of users: 6040 Average actions of users: 138.51266766020865 The number of items: 3629 Average actions of items: 230.56174200661522 The number of inters: 836478 The sparsity of the dataset: 96.18380448885814% Remain Fields: ['user_id', 'item_id', 'timestamp'] 03 Mar 01:41 INFO Build [ModelType.SEQUENTIAL] DataLoader for [train] with format [InputType.PAIRW ISE] 03 Mar 01:41 INFO Evaluation Setting: Group by user_id Ordering: {'strategy': 'by', 'field': ['timestamp'], 'ascending': True} Splitting: {'strategy': 'loo', 'leave_one_num': 2} Negative Sampling: {'strategy': 'by', 'distribution': 'uniform', 'by': 1} 03 Mar 01:41 INFO batch_size = [[1024]], shuffle = [True]

03 Mar 01:41 INFO Build [ModelType.SEQUENTIAL] DataLoader for [evaluation] with format [InputType. POINTWISE] 03 Mar 01:41 INFO Evaluation Setting: Group by user_id Ordering: {'strategy': 'by', 'field': ['timestamp'], 'ascending': True} Splitting: {'strategy': 'loo', 'leave_one_num': 2} Negative Sampling: {'strategy': 'full', 'distribution': 'uniform'} 03 Mar 01:41 INFO batch_size = [[2048, 2048]], shuffle = [False]

03 Mar 01:41 INFO SINE( (loss_fct): CrossEntropyLoss() (C): Embedding(50, 128) (item_embedding): Embedding(3629, 128, padding_idx=0) (ln2): LayerNorm((128,), eps=1e-08, elementwise_affine=True) (ln4): LayerNorm((128,), eps=1e-08, elementwise_affine=True) ) Trainable parameters: 471424 03 Mar 01:41 INFO epoch 0 training [time: 38.95s, train loss: 5937.1911] 03 Mar 01:41 INFO epoch 0 evaluating [time: 0.14s, valid_score: 0.009500] 03 Mar 01:41 INFO valid result: recall@10 : 0.0351 mrr@10 : 0.0095 ndcg@10 : 0.0153 hit@10 : 0.0351 precision@10 : 0.0035

03 Mar 01:41 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:42 INFO epoch 1 training [time: 39.41s, train loss: 5530.0386] 03 Mar 01:42 INFO epoch 1 evaluating [time: 0.14s, valid_score: 0.011600] 03 Mar 01:42 INFO valid result: recall@10 : 0.0460 mrr@10 : 0.0116 ndcg@10 : 0.0195 hit@10 : 0.0460 precision@10 : 0.0046

03 Mar 01:42 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:43 INFO epoch 2 training [time: 37.40s, train loss: 5365.8240] 03 Mar 01:43 INFO epoch 2 evaluating [time: 0.13s, valid_score: 0.014700] 03 Mar 01:43 INFO valid result: recall@10 : 0.0563 mrr@10 : 0.0147 ndcg@10 : 0.0242 hit@10 : 0.0563 precision@10 : 0.0056

03 Mar 01:43 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:43 INFO epoch 3 training [time: 37.27s, train loss: 5260.8537] 03 Mar 01:43 INFO epoch 3 evaluating [time: 0.13s, valid_score: 0.017700] 03 Mar 01:43 INFO valid result: recall@10 : 0.0654 mrr@10 : 0.0177 ndcg@10 : 0.0286 hit@10 : 0.0654 precision@10 : 0.0065

03 Mar 01:43 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:44 INFO epoch 4 training [time: 37.22s, train loss: 5185.0887] 03 Mar 01:44 INFO epoch 4 evaluating [time: 0.13s, valid_score: 0.018800] 03 Mar 01:44 INFO valid result: recall@10 : 0.0704 mrr@10 : 0.0188 ndcg@10 : 0.0306 hit@10 : 0.0704 precision@10 : 0.0070

03 Mar 01:44 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:45 INFO epoch 5 training [time: 37.08s, train loss: 5131.1193] 03 Mar 01:45 INFO epoch 5 evaluating [time: 0.13s, valid_score: 0.019900] 03 Mar 01:45 INFO valid result: recall@10 : 0.0720 mrr@10 : 0.0199 ndcg@10 : 0.0319 hit@10 : 0.0720 precision@10 : 0.0072

03 Mar 01:45 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:45 INFO epoch 6 training [time: 37.03s, train loss: 5089.3843] 03 Mar 01:45 INFO epoch 6 evaluating [time: 0.13s, valid_score: 0.021300] 03 Mar 01:45 INFO valid result: recall@10 : 0.0765 mrr@10 : 0.0213 ndcg@10 : 0.0340 hit@10 : 0.0765 precision@10 : 0.0077

03 Mar 01:45 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:46 INFO epoch 7 training [time: 36.81s, train loss: 5055.0149] 03 Mar 01:46 INFO epoch 7 evaluating [time: 0.13s, valid_score: 0.022400] 03 Mar 01:46 INFO valid result: recall@10 : 0.0802 mrr@10 : 0.0224 ndcg@10 : 0.0357 hit@10 : 0.0802 precision@10 : 0.0080

03 Mar 01:46 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:46 INFO epoch 8 training [time: 37.02s, train loss: 5029.0937] 03 Mar 01:47 INFO epoch 8 evaluating [time: 0.13s, valid_score: 0.022900] 03 Mar 01:47 INFO valid result: recall@10 : 0.0815 mrr@10 : 0.0229 ndcg@10 : 0.0363 hit@10 : 0.0815 precision@10 : 0.0081

03 Mar 01:47 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:47 INFO epoch 9 training [time: 37.11s, train loss: 5007.2925] 03 Mar 01:47 INFO epoch 9 evaluating [time: 0.13s, valid_score: 0.022900] 03 Mar 01:47 INFO valid result: recall@10 : 0.0795 mrr@10 : 0.0229 ndcg@10 : 0.0359 hit@10 : 0.0795 precision@10 : 0.0079

03 Mar 01:48 INFO epoch 10 training [time: 37.46s, train loss: 4988.1709] 03 Mar 01:48 INFO epoch 10 evaluating [time: 0.12s, valid_score: 0.024200] 03 Mar 01:48 INFO valid result: recall@10 : 0.0828 mrr@10 : 0.0242 ndcg@10 : 0.0376 hit@10 : 0.0828 precision@10 : 0.0083

03 Mar 01:48 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:48 INFO epoch 11 training [time: 37.08s, train loss: 4971.1123] 03 Mar 01:48 INFO epoch 11 evaluating [time: 0.30s, valid_score: 0.023100] 03 Mar 01:48 INFO valid result: recall@10 : 0.0810 mrr@10 : 0.0231 ndcg@10 : 0.0364 hit@10 : 0.0810 precision@10 : 0.0081

03 Mar 01:49 INFO epoch 12 training [time: 37.07s, train loss: 4955.7578] 03 Mar 01:49 INFO epoch 12 evaluating [time: 0.13s, valid_score: 0.023500] 03 Mar 01:49 INFO valid result: recall@10 : 0.0805 mrr@10 : 0.0235 ndcg@10 : 0.0365 hit@10 : 0.0805 precision@10 : 0.0080

03 Mar 01:50 INFO epoch 13 training [time: 36.85s, train loss: 4941.3347] 03 Mar 01:50 INFO epoch 13 evaluating [time: 0.13s, valid_score: 0.024700] 03 Mar 01:50 INFO valid result: recall@10 : 0.0858 mrr@10 : 0.0247 ndcg@10 : 0.0387 hit@10 : 0.0858 precision@10 : 0.0086

03 Mar 01:50 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:50 INFO epoch 14 training [time: 37.74s, train loss: 4929.1814] 03 Mar 01:50 INFO epoch 14 evaluating [time: 0.13s, valid_score: 0.024300] 03 Mar 01:50 INFO valid result: recall@10 : 0.0851 mrr@10 : 0.0243 ndcg@10 : 0.0382 hit@10 : 0.0851 precision@10 : 0.0085

03 Mar 01:51 INFO epoch 15 training [time: 36.96s, train loss: 4917.8962] 03 Mar 01:51 INFO epoch 15 evaluating [time: 0.13s, valid_score: 0.024900] 03 Mar 01:51 INFO valid result: recall@10 : 0.0845 mrr@10 : 0.0249 ndcg@10 : 0.0386 hit@10 : 0.0845 precision@10 : 0.0084

03 Mar 01:51 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:51 INFO epoch 16 training [time: 38.13s, train loss: 4907.1214] 03 Mar 01:52 INFO epoch 16 evaluating [time: 0.13s, valid_score: 0.024600] 03 Mar 01:52 INFO valid result: recall@10 : 0.0871 mrr@10 : 0.0246 ndcg@10 : 0.0390 hit@10 : 0.0871 precision@10 : 0.0087

03 Mar 01:52 INFO epoch 17 training [time: 37.74s, train loss: 4897.3860] 03 Mar 01:52 INFO epoch 17 evaluating [time: 0.13s, valid_score: 0.025200] 03 Mar 01:52 INFO valid result: recall@10 : 0.0860 mrr@10 : 0.0252 ndcg@10 : 0.0391 hit@10 : 0.0860 precision@10 : 0.0086

03 Mar 01:52 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:53 INFO epoch 18 training [time: 38.94s, train loss: 4888.8084] 03 Mar 01:53 INFO epoch 18 evaluating [time: 0.13s, valid_score: 0.023500] 03 Mar 01:53 INFO valid result: recall@10 : 0.0821 mrr@10 : 0.0235 ndcg@10 : 0.0370 hit@10 : 0.0821 precision@10 : 0.0082

03 Mar 01:53 INFO epoch 19 training [time: 37.60s, train loss: 4881.1771] 03 Mar 01:53 INFO epoch 19 evaluating [time: 0.13s, valid_score: 0.025200] 03 Mar 01:53 INFO valid result: recall@10 : 0.0883 mrr@10 : 0.0252 ndcg@10 : 0.0396 hit@10 : 0.0883 precision@10 : 0.0088

03 Mar 01:54 INFO epoch 20 training [time: 37.10s, train loss: 4874.0367] 03 Mar 01:54 INFO epoch 20 evaluating [time: 0.13s, valid_score: 0.025600] 03 Mar 01:54 INFO valid result: recall@10 : 0.0911 mrr@10 : 0.0256 ndcg@10 : 0.0406 hit@10 : 0.0911 precision@10 : 0.0091

03 Mar 01:54 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:55 INFO epoch 21 training [time: 37.90s, train loss: 4867.2784] 03 Mar 01:55 INFO epoch 21 evaluating [time: 0.13s, valid_score: 0.024500] 03 Mar 01:55 INFO valid result: recall@10 : 0.0866 mrr@10 : 0.0245 ndcg@10 : 0.0388 hit@10 : 0.0866 precision@10 : 0.0087

03 Mar 01:55 INFO epoch 22 training [time: 38.58s, train loss: 4861.6909] 03 Mar 01:55 INFO epoch 22 evaluating [time: 0.13s, valid_score: 0.026200] 03 Mar 01:55 INFO valid result: recall@10 : 0.0898 mrr@10 : 0.0262 ndcg@10 : 0.0408 hit@10 : 0.0898 precision@10 : 0.0090

03 Mar 01:55 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:56 INFO epoch 23 training [time: 37.71s, train loss: 4854.7629] 03 Mar 01:56 INFO epoch 23 evaluating [time: 0.13s, valid_score: 0.025800] 03 Mar 01:56 INFO valid result: recall@10 : 0.0893 mrr@10 : 0.0258 ndcg@10 : 0.0404 hit@10 : 0.0893 precision@10 : 0.0089

03 Mar 01:57 INFO epoch 24 training [time: 37.22s, train loss: 4849.6407] 03 Mar 01:57 INFO epoch 24 evaluating [time: 0.13s, valid_score: 0.027800] 03 Mar 01:57 INFO valid result: recall@10 : 0.0927 mrr@10 : 0.0278 ndcg@10 : 0.0427 hit@10 : 0.0927 precision@10 : 0.0093

03 Mar 01:57 INFO Saving current best: saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 01:57 INFO epoch 25 training [time: 37.53s, train loss: 4843.9055] 03 Mar 01:57 INFO epoch 25 evaluating [time: 0.13s, valid_score: 0.025600] 03 Mar 01:57 INFO valid result: recall@10 : 0.0918 mrr@10 : 0.0256 ndcg@10 : 0.0407 hit@10 : 0.0918 precision@10 : 0.0092

03 Mar 01:58 INFO epoch 26 training [time: 38.29s, train loss: 4837.8138] 03 Mar 01:58 INFO epoch 26 evaluating [time: 0.13s, valid_score: 0.025500] 03 Mar 01:58 INFO valid result: recall@10 : 0.0898 mrr@10 : 0.0255 ndcg@10 : 0.0403 hit@10 : 0.0898 precision@10 : 0.0090

03 Mar 01:58 INFO epoch 27 training [time: 37.03s, train loss: 4834.1410] 03 Mar 01:58 INFO epoch 27 evaluating [time: 0.13s, valid_score: 0.026300] 03 Mar 01:58 INFO valid result: recall@10 : 0.0921 mrr@10 : 0.0263 ndcg@10 : 0.0414 hit@10 : 0.0921 precision@10 : 0.0092

03 Mar 01:59 INFO epoch 28 training [time: 36.50s, train loss: 4829.7949] 03 Mar 01:59 INFO epoch 28 evaluating [time: 0.13s, valid_score: 0.026000] 03 Mar 01:59 INFO valid result: recall@10 : 0.0913 mrr@10 : 0.0260 ndcg@10 : 0.0409 hit@10 : 0.0913 precision@10 : 0.0091

03 Mar 02:00 INFO epoch 29 training [time: 36.90s, train loss: 4825.3108] 03 Mar 02:00 INFO epoch 29 evaluating [time: 0.13s, valid_score: 0.025400] 03 Mar 02:00 INFO valid result: recall@10 : 0.0888 mrr@10 : 0.0254 ndcg@10 : 0.0399 hit@10 : 0.0888 precision@10 : 0.0089

03 Mar 02:00 INFO epoch 30 training [time: 37.00s, train loss: 4821.6994] 03 Mar 02:00 INFO epoch 30 evaluating [time: 0.13s, valid_score: 0.025400] 03 Mar 02:00 INFO valid result: recall@10 : 0.0908 mrr@10 : 0.0254 ndcg@10 : 0.0403 hit@10 : 0.0908 precision@10 : 0.0091

03 Mar 02:00 INFO Finished training, best eval result in epoch 24 03 Mar 02:00 INFO Loading model structure and parameters from saved/SINE-Mar-03-2022_01-41-19.pth 03 Mar 02:00 INFO best valid result: {'recall@10': 0.0927, 'mrr@10': 0.0278, 'ndcg@10': 0.0427, 'h it@10': 0.0927, 'precision@10': 0.0093} 03 Mar 02:00 INFO test result: {'recall@10': 0.0841, 'mrr@10': 0.0253, 'ndcg@10': 0.0388, 'hit@10' : 0.0841, 'precision@10': 0.0084} 03 Mar 02:00 INFO total training and evaluating time: 1168.7043371200562 03 Mar 02:00 INFO avg time of training an epoch: 37.50458531994973 03 Mar 02:00 INFO inference time: 0.1273343563079834

注：以上结果只设置训练负样本数为1，但原始论文中设置训练负样本数为5，因此我也在最新版本的Recbole中，将其设置为5， neg_sampling: uniform: 5 其余参数与上面的设置一致，均为论文中关于该数据集的最优参数，但运行结果依旧异常，以下为新版本Recbole中运行该模型的输出信息： 03 Mar 02:19 INFO
General Hyper Parameters: gpu_id = 0 use_gpu = True seed = 2020 state = INFO reproducibility = True data_path = dataset/ml-1m checkpoint_dir = saved show_progress = False save_dataset = False dataset_save_path = None save_dataloaders = False dataloaders_save_path = None log_wandb = False

Training Hyper Parameters: epochs = 500 train_batch_size = 1024 learner = adam learning_rate = 0.001 neg_sampling = {'uniform': 5} eval_step = 1 stopping_step = 5 clip_grad_norm = None weight_decay = 0.0 loss_decimal_place = 4

Evaluation Hyper Parameters: eval_args = {'split': {'LS': 'valid_and_test'}, 'order': 'TO', 'mode': 'full', 'group_by': 'user'} repeatable = True metrics = ['Recall', 'MRR', 'NDCG', 'Hit', 'Precision'] topk = [10] valid_metric = MRR@10 valid_metric_bigger = True eval_batch_size = 2048 metric_decimal_place = 4

Dataset Hyper Parameters: field_separator =
seq_separator =
USER_ID_FIELD = user_id ITEM_ID_FIELD = item_id RATING_FIELD = rating TIME_FIELD = timestamp seq_len = None LABEL_FIELD = label threshold = None NEGPREFIX = neg load_col = {'inter': ['user_id', 'item_id', 'rating', 'timestamp']} unload_col = None additional_feat_suffix = None rm_dup_inter = None max_user_inter_num = None min_user_inter_num = None max_item_inter_num = None min_item_inter_num = None lowest_val = {'rating': 3} highest_val = None equal_val = None not_equal_val = None drop_filter_field = True fields_in_same_space = None fill_nan = True preload_weight = None drop_preload_weight = True normalize_field = None normalize_all = True ITEM_LIST_LENGTH_FIELD = item_length LIST_SUFFIX = _list MAX_ITEM_LIST_LENGTH = 20 POSITION_FIELD = position_id HEAD_ENTITY_ID_FIELD = head_id TAIL_ENTITY_ID_FIELD = tail_id RELATION_ID_FIELD = relation_id ENTITY_ID_FIELD = entity_id benchmark_filename = None

Other Hyper Parameters: wandb_project = recbole require_pow = False real_time_process = True embedding_size = 128 layer_norm_eps = 1e-12 prototype_size = 50 interest_size = 4 tau_ratio = 0.1 reg_loss_ratio = 0.5 loss_type = CE SOURCE_ID_FIELD = source_id TARGET_ID_FIELD = target_id MODEL_TYPE = ModelType.SEQUENTIAL MODEL_INPUT_TYPE = InputType.PAIRWISE eval_type = EvaluatorType.RANKING device = cuda train_neg_sample_args = {'strategy': 'by', 'by': 5, 'distribution': 'uniform', 'dynamic': 'none'} eval_neg_sample_args = {'strategy': 'full', 'distribution': 'uniform'}

03 Mar 02:19 INFO ml-1m The number of users: 6041 Average actions of users: 165.5975165562914 The number of items: 3707 Average actions of items: 269.88909875876953 The number of inters: 1000209 The sparsity of the dataset: 95.53358229599758% Remain Fields: ['user_id', 'item_id', 'rating', 'timestamp'] 03 Mar 02:20 INFO [Training]: train_batch_size = [1024] negative sampling: [{'uniform': 5}] 03 Mar 02:20 INFO [Evaluation]: eval_batch_size = [2048] eval_args: [{'split': {'LS': 'validand test'}, 'order': 'TO', 'mode': 'full', 'group_by': 'user'}] 03 Mar 02:20 INFO SINE( (loss_fct): CrossEntropyLoss() (C): Embedding(50, 128) (item_embedding): Embedding(3707, 128, padding_idx=0) (ln2): LayerNorm((128,), eps=1e-12, elementwise_affine=True) (ln4): LayerNorm((128,), eps=1e-12, elementwise_affine=True) ) Trainable parameters: 481408 03 Mar 02:23 INFO epoch 0 training [time: 209.18s, train loss: 34077.1404] 03 Mar 02:23 INFO epoch 0 evaluating [time: 0.07s, valid_score: 0.017300] 03 Mar 02:23 INFO valid result: recall@10 : 0.0627 mrr@10 : 0.0173 ndcg@10 : 0.0277 hit@10 : 0.0627 precision@10 : 0.0063 03 Mar 02:23 INFO Saving current: saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 02:27 INFO epoch 1 training [time: 210.64s, train loss: 31420.3748] 03 Mar 02:27 INFO epoch 1 evaluating [time: 0.07s, valid_score: 0.021400] 03 Mar 02:27 INFO valid result: recall@10 : 0.0747 mrr@10 : 0.0214 ndcg@10 : 0.0336 hit@10 : 0.0747 precision@10 : 0.0075 03 Mar 02:27 INFO Saving current: saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 02:30 INFO epoch 2 training [time: 210.18s, train loss: 30558.1838] 03 Mar 02:30 INFO epoch 2 evaluating [time: 0.07s, valid_score: 0.026100] 03 Mar 02:30 INFO valid result: recall@10 : 0.0894 mrr@10 : 0.0261 ndcg@10 : 0.0406 hit@10 : 0.0894 precision@10 : 0.0089 03 Mar 02:30 INFO Saving current: saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 02:34 INFO epoch 3 training [time: 210.84s, train loss: 30111.9773] 03 Mar 02:34 INFO epoch 3 evaluating [time: 0.09s, valid_score: 0.028100] 03 Mar 02:34 INFO valid result: recall@10 : 0.0925 mrr@10 : 0.0281 ndcg@10 : 0.0428 hit@10 : 0.0925 precision@10 : 0.0093 03 Mar 02:34 INFO Saving current: saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 02:37 INFO epoch 4 training [time: 210.20s, train loss: 29832.5896] 03 Mar 02:37 INFO epoch 4 evaluating [time: 0.08s, valid_score: 0.028300] 03 Mar 02:37 INFO valid result: recall@10 : 0.094 mrr@10 : 0.0283 ndcg@10 : 0.0433 hit@10 : 0.094 precision@10 : 0.0094 03 Mar 02:37 INFO Saving current: saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 02:41 INFO epoch 5 training [time: 210.42s, train loss: 29627.7136] 03 Mar 02:41 INFO epoch 5 evaluating [time: 0.07s, valid_score: 0.028200] 03 Mar 02:41 INFO valid result: recall@10 : 0.0975 mrr@10 : 0.0282 ndcg@10 : 0.0441 hit@10 : 0.0975 precision@10 : 0.0098 03 Mar 02:44 INFO epoch 6 training [time: 210.28s, train loss: 29472.4046] 03 Mar 02:44 INFO epoch 6 evaluating [time: 0.07s, valid_score: 0.027900] 03 Mar 02:44 INFO valid result: recall@10 : 0.095 mrr@10 : 0.0279 ndcg@10 : 0.0432 hit@10 : 0.095 precision@10 : 0.0095 03 Mar 02:48 INFO epoch 7 training [time: 208.80s, train loss: 29358.4611] 03 Mar 02:48 INFO epoch 7 evaluating [time: 0.07s, valid_score: 0.031000] 03 Mar 02:48 INFO valid result: recall@10 : 0.1008 mrr@10 : 0.031 ndcg@10 : 0.047 hit@10 : 0.1008 precision@10 : 0.0101 03 Mar 02:48 INFO Saving current: saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 02:51 INFO epoch 8 training [time: 206.79s, train loss: 29262.8998] 03 Mar 02:51 INFO epoch 8 evaluating [time: 0.07s, valid_score: 0.030200] 03 Mar 02:51 INFO valid result: recall@10 : 0.1012 mrr@10 : 0.0302 ndcg@10 : 0.0464 hit@10 : 0.1012 precision@10 : 0.0101 03 Mar 02:55 INFO epoch 9 training [time: 207.73s, train loss: 29179.4955] 03 Mar 02:55 INFO epoch 9 evaluating [time: 0.07s, valid_score: 0.029700] 03 Mar 02:55 INFO valid result: recall@10 : 0.0977 mrr@10 : 0.0297 ndcg@10 : 0.0452 hit@10 : 0.0977 precision@10 : 0.0098 03 Mar 02:58 INFO epoch 10 training [time: 207.99s, train loss: 29111.3402] 03 Mar 02:58 INFO epoch 10 evaluating [time: 0.07s, valid_score: 0.030200] 03 Mar 02:58 INFO valid result: recall@10 : 0.0997 mrr@10 : 0.0302 ndcg@10 : 0.0461 hit@10 : 0.0997 precision@10 : 0.01 03 Mar 03:02 INFO epoch 11 training [time: 208.41s, train loss: 29044.1697] 03 Mar 03:02 INFO epoch 11 evaluating [time: 0.07s, valid_score: 0.030600] 03 Mar 03:02 INFO valid result: recall@10 : 0.1031 mrr@10 : 0.0306 ndcg@10 : 0.0472 hit@10 : 0.1031 precision@10 : 0.0103 03 Mar 03:05 INFO epoch 12 training [time: 208.34s, train loss: 28996.1376] 03 Mar 03:05 INFO epoch 12 evaluating [time: 0.08s, valid_score: 0.029200] 03 Mar 03:05 INFO valid result: recall@10 : 0.106 mrr@10 : 0.0292 ndcg@10 : 0.0467 hit@10 : 0.106 precision@10 : 0.0106 03 Mar 03:09 INFO epoch 13 training [time: 208.41s, train loss: 28944.6357] 03 Mar 03:09 INFO epoch 13 evaluating [time: 0.07s, valid_score: 0.030100] 03 Mar 03:09 INFO valid result: recall@10 : 0.1036 mrr@10 : 0.0301 ndcg@10 : 0.0469 hit@10 : 0.1036 precision@10 : 0.0104 03 Mar 03:09 INFO Finished training, best eval result in epoch 7 03 Mar 03:09 INFO Loading model structure and parameters from saved/SINE-Mar-03-2022_02-20-10.pth 03 Mar 03:09 INFO best valid : OrderedDict([('recall@10', 0.1008), ('mrr@10', 0.031), ('ndcg@10', 0.047), ('hit@10', 0.1008), ('precision@10', 0.0101)]) 03 Mar 03:09 INFO test result: OrderedDict([('recall@10', 0.096), ('mrr@10', 0.0297), ('ndcg@10', 0.0449), ('hit@10', 0.096), ('precision@10', 0.0096)])

预期 希望针对SINE在ml-1m上的结果异常问题，能够找到可能的原因

Ethan-TZ commented 2 years ago

@Sword-CS 感谢您的关注！ 首先原论文中使用的mini-batch size为128，与你的设置不一致。并且原论文的所有实验是基于tensorflow实现的，pytorch和tensorflow的底层有一些区别，参数初始化以及随机因素导致模型存在差异很正常。 伯乐是一个相对公平的平台来比较不同模型，但考虑到不同模型之间存在着差异，为了建立一个统一的框架，我们需要在各个环节做权衡和调整，不能完全兼顾某一种模型，所以我们的代码开发相比于原文代码，也许会在数据处理等部分有所不同。但是我们可以保证模型之间的排序。

xingyuz233 commented 2 years ago

@chenyuwuxin 我也遇到了同样的问题，即ML-1M上表现跟原文中的指标差异太大。

2017pxy commented 2 years ago

@xingyuz233 @Sword-CS 你好，关于SINE结果在ml-1m差异过大的问题，我们正在检查，之后会在这里进行回复。此外，如果你们发现了我们代码中的bug，也欢迎你给予反馈。感谢你们对伯乐的关注！

cyxg7 commented 1 year ago

@xingyuz233 @Sword-CS 你好，关于SINE结果在ml-1m差异过大的问题，我们正在检查，之后会在这里进行回复。此外，如果你们发现了我们代码中的bug，也欢迎你给予反馈。感谢你们对伯乐的关注！

请问有新的进展了吗，我这边差异还是特别大