New Feature: ignore labels

[comorbidity]

Relates to https://github.com/smart-on-fhir/chart-review/issues/12 https://github.com/smart-on-fhir/chart-review/issues/13

study-specific labels may be ignored per request of the researcher.

For example, when scoring agreement Rena VS Andy, we want to score ideation-past.

However, when scoring agreement of Rena vs ICD10 we want to ignore ideation-past because ICD10 does not have a code for this.

Config changes could enable to switch on/off which labels to use and which labels to ignore. Implementation: could be a rewrite rule, such as when label = X then rewrite as label = None

[mikix]

Hmm... this might need some light thinking - if it's so flexible as you say, a config-file flag might not be the right place for this, but rather a command-line flag.

But on the other hand, maybe that would be annoying/error-prone to specify all the time.

[mikix]

Hmm, OK I can foresee two areas where this might happen:

• A reviewer labeled notes before the final set of notes was defined, and a new label has been added since
• An external source can't provide a label (like ICD10 and past-ideation)

In both cases, it seems like this is per-annotator config (human or external). Here is what our annotator config looks like today:

annotators:
  human1: 2
  human2: 3
  icd10:
    filename: icd10-labels.csv

We have a good place for this to be added for external annotators, since their config is already a dictionary and we can add a new key like ignored-labels. Humans don't have such a good place yet, since they just accept an int. But maybe we can punt on human ignored labels for now, since that use case seems lower priority. And it can easily be implemented in the future with optional dictionary support in the config.

So, I'm proposing an initial MVP version of this feature like:

annotators:
  icd10:
    filename: icd10-labels.csv
    ignored-labels: [ideation-past]

A future version for humans (or I might implement this in the first go, if I'm feeling fancy) might look like:

annotators:
  human1:
    id: 2
    ignored-labels: [ideation-past]

[mikix]

OK we talked and now we're thinking actually, let's allow multiple configs for now. One with the label and one without.

Just because you may very well want two versions of the human reviewers too - one with ideation-past and one without (for better apples-to-apples with icd10 and for a view of a world with that label too).

So then we either allow a CLI option for toggling which mode you're in, use a workaround like two versions of a human defined, with different names, or the option we're going with for now: allow multiple configs.

That might be more broadly useful anyway (like for grouping support in #13)

[mikix]

This user need should be solved by allowing separate configs, landed above ^