search

AmenRa / ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍
https://amenra.github.io/ranx
MIT License
427 stars 23 forks source link

[Feature Request] Add interpolated recall-precision plot function #38

Closed EingeoelterBolt closed 1 year ago

EingeoelterBolt commented 1 year ago

Is your feature request related to a problem? Please describe. First of all: This is a really nice library! It helps a lot! My request is regarding a recall-precision graphic. When I read TREC-related papers, they very often used the interpolated precision-recall-plot to visualize the performance of IR-Systems which are being compared to each other. They also use the graphs to understand which IR-System yields a higher recall value, shows a higher precision value, and generally has a better performance.

Describe the solution you'd like Since I love using this library, it would be great, if there were a function for generating such a precision-recall-plot or just an example/notebook for generating this plot using the ranx library in the documentation.

Describe alternatives you've considered I've already considered creating these plots myself with the functions that the library offers. However, it seems quite complex as the interpolation greatly complicates this work for me.

Additional context Here is an example interpolated recall-precision plot: image

More information can be found on: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html and https://trec.nist.gov/pubs/trec15/appendices/CE.MEASURES06.pdf (p.4)

AmenRa commented 1 year ago

Hi, thanks for the very detailed suggestion/request! I think this would be a nice addition to the library! I will take a look to the linked documents in the coming days.

EingeoelterBolt commented 1 year ago

I found a really nice lecture on youtube, which explains the recall-precision plot really nice. It can be found here:

https://youtu.be/x0SkRpIVMUs?t=1066

I also found out, that the official trec_eval program (https://github.com/usnistgov/trec_eval) returns the values for creating this plot. My workaround is actually, that i create runs using ranx, export the runs to a .trec-file and use the official trec_eval program

AmenRa commented 1 year ago

Hi, sorry for the delay. I finally have a working implementation for computing the graph! I still have to figure out a couple of usability things. You can expect to be able to compute the Precision-Recall graph with ranx soon.

AmenRa commented 1 year ago

Hi, I added the Interpolated Precision-Recall Curve in v0.3.8. I tested my implementation against trec_eval for correctness. You can find a usage example here.

Please, let me know if you have any comment / suggestion.