Develop an R&D tool that can be used to collect labelled search relevance test data

[kdutia]

• [x] develop basic tool
• [x] lit review - think about how best to generate (/sample) search results to create this dataset. see here
• [x] find climate-related queries from msmarco
• [x] come up with and add definitions of result quality measures, based on nDCG
• [x] add user field
• [x] move page selection to sidebar
• [ ] ~enable user to enter custom queries~ my feeling is that we don't do this in test mode, as we want to look at inter-annotator agreement. Waiting for Chris' response
• [x] write storing annotations in JSON

lit review

• sampling techniques
• avoiding bias
• which models are used to generate test datasets

In future we'll need to agree on one use case/information need that we're conducting this test under, as well as the type of search. -> add these notes to the Notion

[kdutia]

Developed a basic UI at https://github.com/climatepolicyradar/search-review-ui

[kdutia]

From a look at the queries in MSMarco they won't be a great source of extra queries for us, as ones about climate tend to be asking quite basic questions about the climate. These wouldn't fit with our user profiles. Examples:

[kdutia]

The first version of this tool, suitable for reviewing, is largely complete. To improve and extend it, I'd suggest:

• Adding a timestamp to each annotation set that is saved to disk.
• Creating a batch job which runs all the test queries through a given search approach and returns the top K (=50? =100?) results. These results can then be randomly sampled to show to different users. This has the benefits of speeding up the working of the tool, allowing it to run on a very small EC2 machine, and ensuring reproducibility of search results across different users.
• Sampling and shuffling the cached results from the last point. Each user should be given a different set of results, shuffled - but there should be overlap between results served to different people, so that we can measure inter-annotator agreement. Sample e.g. 20 of 40?
• Creating a more generic approach of a query/search for the utils.do_search function, so that we can extend the experimenting side of this tool to allow users to easily select different search engine variants to play with.

@chrisaballard - could we try to find half an hour for a chat about the current state of this and the next steps this week? My suggestion is that we then raise another issue to finish off the work next sprint. We'll then have a tool which both allows us to label data and a basis of a tool which allows us to easily experiment with different search engines.