search

agiresearch / OpenP5

OpenP5: An Open-Source Platform for Developing, Training, and Evaluating LLM-based Recommender Systems
Apache License 2.0
253 stars 20 forks source link

Wrong results for LastFM-Collaborative indexing #32

Open mgulcin opened 6 months ago

mgulcin commented 6 months ago

I'm trying to replicate the results from the paper (on LastFm dataset).

Even if I managed to replicate results for 'random' and 'sequential' indexing, I couldn't replicate results for 'collaborative' (OpenP5-T5-C (un/seen)).

Here is an example result I obtained, using same command and model (from checkpoints provided): OpenP5-T5-C (unseen)

testing filtered LastFM dataset on sequential task
hit@5: 0.0027522935779816515
hit@10: 0.0027522935779816515
ndcg@5: 0.001392461064959632
ndcg@10: 0.001392461064959632

whereas results in the paper:

hit@5: 0.0504 
ndcg@5: 0.0332 
hit@10: 0.0724 
ndcg@10:0.0420

Can you please guide me with right command or right model (checkpoint)?

xiaxin1998 commented 1 month ago

same problem here. I used the command in the repo.