New Feature: Aggreagate multiple annotators into a set of ground-truth annotations

[Dtphelan1]

Motivation: Often multiple annotators divide and conquer for a single annotation task. The combination of their annotations are then used as a ground truth against which we compare some NLP/LLM system's output.

To accommodate this workflow – compare one set of output against multiple annotators ground truth – there should be some way of specifying a set of multiple annotators as a single "ground truth" annotator.

One approach might be to support this at the config level; similar to grouped-labels, we could introduce something like grouped-annotators, allowing for the aliasing of multiple annotators under a single, new annotator id.

Another approach might be to take this from the command-level; create some flexibility on the CLI side to support specifying multiple annotators for the ground-truth argument.

[mikix]

Great point. This is a useful feature that we don't have right now. Andy had written something like this for a covid paper, but I had questions about how to accurately generalize what he did, without introducing new math bugs of my own. :smile:

This is the old code he wrote, FYI.

My gut prefers the CLI approach - like just specify multiple annotators on the CLI and this mode would kick in.