smart-on-fhir / chart-review

Measure agreement between chart reviewers.
https://docs.smarthealthit.org/cumulus/chart-review/
Apache License 2.0
3 stars 1 forks source link

Consider adding a way to export Label Studio labels #39

Closed mikix closed 3 months ago

mikix commented 3 months ago

It has occasionally happened that an export of the labels in Label Studio would be helpful, ala an "external" set of CSV annotations.

Something like:

chart-review export-ls-labels <annotator>

And you get a CSV of note_id,label for that annotator, like an external annotation.

This is not widely useful, but could be helpful for some fancy post-processing a researcher might do. I think the last time it came up for BCH was when we were playing with comparing some LLM results against human annotations. Just recording this idea here, so we have a place to track the request.

(This issue is split off from #26)

comorbidity commented 3 months ago

This is not widely useful, but could be helpful for some fancy post-processing a researcher might do

👋 Hi, researcher here this IS widely useful :) Came here to request this exact feature. Additional columns of interest are which annotator (chart-reviewer) and the text selected

Ideal case: note_id, annotator_id, label, mention

mikix commented 3 months ago

Can you talk more about the use case where you wanted this?

Adding mention text and annotators is a bit above and beyond the use case we had hit originally, where we really just wanted the ability to unify "note/label" data -- specifically, to format Label Studio labels as if they were an external annotation set with two columns of docref_id,label.

So there might be two different use cases here and I want to be sure we don't build the wrong thing.

mikix commented 3 months ago

Andy mentioned this request on slack, which might be feature-related or not, depending:

Summarize frequency of annotation at different scale/resolutions.