search

khui / copacrr

The code for COPACRR Neural IR model.
Apache License 2.0
38 stars 11 forks source link

More of a doubt than an issue, regarding evaluation #4

Closed JoaoLages closed 6 years ago

JoaoLages commented 6 years ago

Hi there!

I have been trying to understand where do the files under data/trec_runs/wt**/ come from. I thought they would be an intersection between this QL submissions from TREC and the qrel files from that year, but I haven't been able to reproduce what you have. Maybe I am confused and this files are actually for the RERANK-ALL rather than the RERANK evaluation.

andrewyates commented 6 years ago

Yep, these files are actually used by RERANK-ALL. I believe the current cwindri files come from querying the Indri ClueWeb Batch API. To compare against the submitted TREC runs, those runs need to be downloaded from TREC and placed in the appropriate directories.

JoaoLages commented 6 years ago

Hmm.. What do these files represent then? I thought RERANK-ALL would be the same as RERANK, but for all the submissions of that year. In fact, is the code for RERANK available? As I don't see the QL files for each year.

andrewyates commented 6 years ago

Yes, you're right that RERANK-ALL is RERANK over all TREC runs from a given year.

I just looked through the eval code again and realized my last comment was incorrect. If you use the current cwindri files with evals/rerank.py, you should get the RERANK evaluation. If you put the TREC run files in that directory and run rerank.py, you get the RERANK-ALL evaluation.

JoaoLages commented 6 years ago

Ok, that's what I understood. So if cwindri gives you RERANK, cwindri are the QL baseline submissions (from years 12-14), right?

If that's correct, I was searching for the origin of those cwindri QL submissions, I only find the "indri" ones, from this link.

On 23 Nov 2017 5:40 p.m., "Andrew Yates" notifications@github.com wrote:

Yes, you're right that RERANK-ALL is RERANK over all TREC runs from a given year.

I just looked through the eval code again and realized my last comment was incorrect. If you use the current cwindri files with evals/rerank.py, you should get the RERANK evaluation. If you put the TREC run files in that directory and run rerank.py, you get the RERANK-ALL evaluation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/khui/repacrr/issues/4#issuecomment-346673329, or mute the thread https://github.com/notifications/unsubscribe-auth/AQwpDbIhbGdM7YT9bBP8HCgqwxfBXTMnks5s5a4ogaJpZM4Qo1d7 .

andrewyates commented 6 years ago

Right, cwindri are Indri QL baseline submissions. The source of those cwindri files is the ClueWeb Batch Query Service.

In the PACRR and RE-PACRR papers, however, we used the Terrier QL results from the link you gave. I guess we switched to using cwindri in the repo because we also had those files for 2009-2011.

JoaoLages commented 6 years ago

Nice, mystery solved! So, just to make sure, can you confirm me if you reported results in the paper using ql/results-cata.txt from this link? This is from 2014, but the same applies to 2012/13. It's mainly because I want to evaluate in the exact same way and be able to use your previously reported results, so that I don't have to run DUET, DRMM, etc :)

andrewyates commented 6 years ago

Yep, that's right!

JoaoLages commented 6 years ago

Ok, cool. I will just grab those files, intersect them with the QREL_adhoc pairs from each respective earl and rerank them :)

I will close the issue, thanks for the clarification!

On 24 Nov 2017 1:37 p.m., "Andrew Yates" notifications@github.com wrote:

Yep, that's right!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/khui/repacrr/issues/4#issuecomment-346830374, or mute the thread https://github.com/notifications/unsubscribe-auth/AQwpDaCS27KHWH5YDkd2gb4qpYd_hIN_ks5s5saMgaJpZM4Qo1d7 .