sara-chronister / syndrome-definition-evaluation

R code for evaluation of NSSP BioSense ESSENCE syndrome definition results using ESSENCE APIs.
13 stars 5 forks source link

Add inter rater tools #15

Closed DOH-SNC1303 closed 6 months ago

DOH-TJB0303 commented 11 months ago

Add tool that will support inter-rater/consensus manual review of a sample of pulled records.

DOH-TJB0303 commented 10 months ago

Additional Checklist of Updates:

DOH-TJB0303 commented 10 months ago

We are factoring Review Category Labels (True Positive, False Positive) before feeding the reviewer data to caret::confusionMatrix(). Under this format, "Uncertain" Review Category labels are set to NA within the factor, and I believed dropped from the query metric calculations.

@DOH-SNC1303 @DOH-SNC1303

Question: Is there a way to incorporate "Uncertain' review labels into Query performance metrics? I have really only seen 2x2 (TP/FP and TN/FN) so dropping "Uncertain" review categories seems like an ok approach...

DOH-TJB0303 commented 10 months ago

Considerations for additional SRS variables:

DOH-TJB0303 commented 9 months ago

12/14/2023:

Added # Create Matched Elements Subfolder fs::dir_create(paste0(output_folder, "Matched_Elements"))

to support Matched_Elements subfolder creation (where data is written)