search

hpclab / rankeval

Official repository of RankEval: An Evaluation and Analysis Framework for Learning-to-Rank Solutions.
http://rankeval.isti.cnr.it/
Mozilla Public License 2.0
88 stars 11 forks source link

wishlist: TF-Ranking support #15

Open zenogantner opened 5 years ago

zenogantner commented 5 years ago

It would be great to have support for the (fairly new) TF-Ranking library by Google: https://github.com/tensorflow/ranking

strani commented 5 years ago

I'm playing right now with tf-ranking in order to compare its performance with that of state-of-the-art ensemble-based models.

That said, how do you think RankEval can support a neural network model given it is based on ensemble of regression trees? The simplest way to "integrate" it would be to extend the tf model with a score method with the same signature of the score method in the RTEnsemble class in such a way the analysis methods can be used without incurring in errors. Obviously, it means we will have a dependency with tf and tf-ranking for such models.

If you have any other suggestion, please let me know.

strani commented 5 years ago

Another option could be to fork the RTEnsemble class, creating a NNModel, with a common Model base class. All the utility methods (the one of course for which it makes sense to accept different models, e.g., not the topological analysis) will accept a base Model instance, and the signatures of the methods belonging to the two classes should be identical. This way everything will be transparent to the user, but still we need the dependency with tf and tf-ranking.

This is probably a cleaner solution, but with more code to change.

zenogantner commented 5 years ago

I like the common base class approach. The dependency does not have to be a hard one, the import could happen on demand.

strani commented 5 years ago

Let's start thus implementing this solution. I'm not sure about the timing of the implementation, I'm busy till next week then could start working on it.

strani commented 5 years ago

Update: I'm quite overwhelmed by the work this week and the next one. The implementation will not start before the middle of February...