search

o19s / quepid

Improve your Elasticsearch, OpenSearch, Solr, Vectara, Algolia and Custom Search search quality.
http://www.quepid.com
Apache License 2.0
284 stars 101 forks source link

Judgement Book configured to support "up to three ratings" per query-doc pair collects up to as many as judges are working on the Book #1019

Closed wrigleyDan closed 5 months ago

wrigleyDan commented 5 months ago

Describe the bug When I define a Book to support multiple raters all raters can give a judgement for a query-doc pair. In the selection strategy it clearly states "up to three", so I report this as a bug.

To Reproduce Steps to reproduce the behavior:

  1. In a Quepid case create a book with the Selection strategy "Multiple Raters"
  2. Make sure to share the book with a team that consists of four or more persons/accounts
  3. Populate the book with query/doc pairs
  4. To restrict the number of judgements you can edit the book and cut off everything after position 1
  5. Make all judges collect some judgements
  6. Once every judge has rated all query/doc pairs you will see that query/doc pairs are not restricted to three judgements.

Expected behavior Only select query/doc pairs to be judged when this query/doc pair has less than three ratings.

Screenshots n/a

Additional context n/a