Reranking Evaluation

[tuongtranngoc]

@nianlonggu How to evaluate only reranking module without running pre-fetched candidate?

[nianlonggu]

You can use the file test_with_oracle_prefetched_ids_for_reranking.json in the shared dataset on google drive. It is a list of Json string, each line contain a Json string of a data example. The "prefetched_ids" contains 2000 BM25 prefetched candidates and the actually cited id "positive_id" is also manually added into the prefetched_ids list. With this file you do not need to run the prefetch, but just run the reranker as shown at https://github.com/nianlonggu/Local-Citation-Recommendation?tab=readme-ov-file#get-paper-recommendations

[tuongtranngoc]

Thanks @nianlonggu , I used the json that you mentioned, but I got error when loading it: json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 24088) Can you help me check it?

[nianlonggu]

You can read the file line by line, and convert each line to a dictionary by json.loads()

[tuongtranngoc]

Thanks @nianlonggu, I got it So, how to produce reranking performance as below results:

I do not find guideline in README

[nianlonggu]

You can refer to the last section in https://github.com/nianlonggu/Local-Citation-Recommendation/blob/main/Turorial_Local_Citation_Recommendation.ipynb

[tuongtranngoc]

Sorry @nianlonggu , as I know, the last section is evaluated entire Recommendation Pipeline both prefetched and reranking. I only wanna to evaluate reranker that you mentioned in paper with table 3

[nianlonggu]

Yes. But you can easily customize it by replacing the prefetched IDs with the "prefetched_ids" I mentioned in the previous Json file.

[tuongtranngoc]

thanks @nianlonggu , I got it