search

Gu-Youngfeng / ReConfigSRC

ReConfig is a post-processing approach to improve the ranking accuracy of the rank-based approach.
1 stars 0 forks source link

[ERROR]:Error in AdaRank::load(): #5

Open Gu-Youngfeng opened 4 years ago

Gu-Youngfeng commented 4 years ago

Last commit a62edaf solved the issue #4 successfully, but it simultaneously triggered another error when running reconfig.py using ranker AdaRank, look at the following failing trace,

Discard orig. features
Model file:     trainer.txt
Feature normalization: No
Model:          AdaRank
Exception in thread "main" ciir.umass.edu.utilities.RankLibError: Error in AdaRank::load():
        at ciir.umass.edu.utilities.RankLibError.create(RankLibError.java:34)
        at ciir.umass.edu.learning.boosting.AdaRank.loadFromString(AdaRank.java:347)
        at ciir.umass.edu.learning.RankerFactory.loadRankerFromString(RankerFactory.java:109)
        at ciir.umass.edu.learning.RankerFactory.loadRankerFromFile(RankerFactory.java:99)
        at ciir.umass.edu.eval.Evaluator.rank(Evaluator.java:1258)
        at ciir.umass.edu.eval.Evaluator.main(Evaluator.java:527)
Caused by: java.lang.NullPointerException
        at ciir.umass.edu.learning.boosting.AdaRank.loadFromString(AdaRank.java:333)
        ... 4 more

There is a possibility that some parameters in AdaRank are mis-assignmented. Here are the commands we used to invoke ranker AdaRank. The error happened when running command 2.

Command 1: Training a model trainer.txt based on trainset_txt15.txt.

>> java -jar RankLib.jar -train trainset_txt15.txt -ranker 3 -metric2t ERR@10 -round 500 -tolerance 0.002 -max 5 -save trainer.txt

Command 2: Re-ranking the testset_txt15.txt based on trainer.txt, then save the results in result.txt

>> java -jar RankLib.jar -rank testset_txt15.txt -load trainer.txt -indri result.txt
Gu-Youngfeng commented 4 years ago

After tried many times to tune the parameters, I found that if I change the -metric2t ERR@10 to -metric2t NDCG@10, the above error will be clear. However, the re-ranked list is the same with the original list. That is, AdaRank CANNOT change the relative order of configurations in the original list.

Gu-Youngfeng commented 4 years ago

Maybe all the metrics to be optimized in a learning-to-rank model should be set to -metric2t NDCG@k. That is, the command 1 should be,

>> java -jar RankLib.jar -train train_set.txt -ranker XX -metric NDCG@10 -save trainer.txt