Run from scratch without preprocessed data get the different result

[Aidenzich]

Sorry to bother you again, I tried to run from scratch without preprocessed data you provided (beauty dataset), but get a different result, I check the user_num, item_num, and total transaction number in new preprocessed data are totally the same as the data you provided, but the performance is different:

• Using the preprocessed data you provided (which is similar to the score in the paper):

  {
    "Recall@1": 0.02538818603550846,
    "Recall@10": 0.08738722245801579,
    "Recall@20": 0.1217391010035168,
    "NDCG@1": 0.02538818603550846,
    "NDCG@10": 0.052028932799161834,
    "NDCG@20": 0.060672693737697875
  }

• Delete the preprocessed data and then run from scratch with python main.py --template train_bert --dataset_code beauty:

  {
    "Recall@1": 0.02049122752875767,
    "Recall@10": 0.07811134181578051,
    "Recall@20": 0.11032424613156101,
    "NDCG@1": 0.02049122752875767,    
    "NDCG@10": 0.04529115934432908,    
    "NDCG@20": 0.053376406177201054    
  }

Are there any differences in your pre-processing methods? If you know how to solve this issue, please kindly advise. Thank you.

[hw-du]

The dataset for Beauty and Toys are copied from https://github.com/aHuiWang/CIKM2020-S3Rec/tree/master/data and converted into pickle format. There might be some problems (due to instable sorting it might disorder the sequence) with the dataprocessing program from https://github.com/jaywonchung/BERT4Rec-VAE-Pytorch, so we do not use this datapreprocessing program.

[Aidenzich]

Understood, thanks for the response!