search

aliannejadi / qulac

Qulac: A dataset on asking Questions for Lack of Clarity in open-domain information-seeking conversations.
MIT License
73 stars 9 forks source link

Meaning of QL output #8

Open zhenduow opened 1 year ago

zhenduow commented 1 year ago

Hi,

Thank you for putting together this amazing instruction. It is easy to follow and works smooth. I have no problem getting the output in run_QL.ipynb. However, I find them hard to understand:

  1. I get the middle column is the ranking score of some sort.
  2. The first column seems like the clueweb09 index and the only information source for computing ranking performance, but how do I get the relevance labels of them for the query 1-1-1?

Can you please explain the output dataframe, especially what the columns mean and how to compute the ranking metrics?

Thank you very much!

aliannejadi commented 1 year ago

Hi, Thank you for your questions and interest.

  1. yes that is the ranking.
  2. The qrel can be made easily from the qrels published by TREC Web track. For each query, you have the topic and facet ID and can use that to get the relevance judgements of TREC Web track.

With the qrels, you can easily compute the metrics using trec_eval or pytrec_eval.

Hope this helps, Mohammad