search

jcastro-inf / delfos

delfos: DEveLopers' Framework of Suggestions is a recommender systems library that has a number of recommendation algorithms, datasets, experimentation protocols and metrics to perform experiments to evaluate and compare recommendation techniques.
2 stars 1 forks source link

GroupCaseStudy replicability error #1

Open jcastro-inf opened 8 years ago

jcastro-inf commented 8 years ago

When a GroupCaseStudy xml is re-executed, the replicability is ok. However, when the xml is re-generated, even if the content is the same, it produces different recommendation results. This is a serious issue, because other researchers that use delfos in the future might obtain different results.

jcastro-inf commented 8 years ago

Check if a re-generated XML has differences with the original one. Check specially seeds settings.

jcastro-inf commented 8 years ago

This issue might be related with the validation (train-test of a dataset.). The replicability of the validation should be tested.

jcastro-inf commented 8 years ago

The replicability error is produced by each group formation technique. Each class needs to be evaluated individually in order to fix this bug.