autotune Function Usage Example

[msharara1998]

I am looking for an example of how to structure the queries and qrels parameters of the autotune function because I searched in the repo and didn't find any example for that. Precisely, what should be the keys and values of queries dict? and similarly for qrels dict?

Thanks in advance for your help.

[AmenRa]

Hi, you can find a very condensed example here.

queries is a list of dictionaries where each dictionary has two keys: q_id and text. Both q_id and text must be strings.

qrels is a dictionary of dictionaries. The parent dictionary keys must be the q_id of the dictionaries in the queries list. Each child dictionary has a key for each document relevant to the corresponding query. The values corresponding to the keys are the relevance score judgments (i.e., how well the document answer the query). Note that they can be both graded or binary (zero means not relevant). Most metrics for performance evaluation of search engines consider all non-judged documents as non-relevants. Therefore, you are good by providing scores only for the documents you know are relevant.

Example of qrels:

qrels = { "q_1": { "d_12": 5, "d_25": 3 },
          "q_2": { "d_11": 6, "d_22": 1 } }

Let me know if it is clear. :)