Update Bulk Result File Validation: Result Code

DanielSass commented 1 month ago

File validation for the bulk result uploader was a contributing factor to the bug described in #7897.

Requested update:

Adjust the validation for the test_result field in the bulk uploader to be more stringent.

/backend/src/main/java/gov/cdc/usds/simplereport/validators/CsvValidatorUtils.java#L273-L275

  public static List<FeedbackMessage> validateTestResult(ValueOrError input) {
    return validateSpecificValueOrSNOMED(input, TEST_RESULT_VALUES);
  }

Rather than allowing anything that looks like a snomed (i.e. the regex currently in place), we should validate against a list of specific snomeds that we support and can effectively map in the fhir converter (which should include any snomeds in the covid or hiv livd tables).

The SNOMEDs we allow should be the same ones being explicitly mapped to normal/abnormal flags in #7897

emyl3 commented 1 month ago

As part of this we should ensure the following: We should make sure that any results being sent to the CSV pipeline should also map to values here: https://github.com/CDCgov/prime-reportstream/blob/1ffae4ca0b04cd0aa9f169e26813ecd86df71bb5/prime-router/src/main/kotlin/metadata/Mappers.kt#L768 otherwise you get the following error from ReportStream that is not obvious to users uploading their results: RS CSV API Error 400 Response:

{
  "id" : null,
  "submissionId" : 4261955,
  "overallStatus" : "Error",
  "timestamp" : "2024-07-11T20:43:28.954Z",
  "plannedCompletionAt" : null,
  "actualCompletionAt" : null,
  "sender" : "",
  "reportItemCount" : null,
  "errorCount" : 1,
  "warningCount" : 0,
  "httpStatus" : 400,
  "destinations" : [ ],
  "actionName" : "receive",
  "externalName" : "",
  "reportId" : null,
  "topic" : null,
  "bodyFormat" : "",
  "errors" : [ {
    "scope" : "item",
    "indices" : [ 1 ],
    "trackingIds" : [ "42daf9c8-9deb-4636-a040-380bde2b44cd+1+20240711" ],
    "field" : "test_result (test_result)",
    "message" : "The code '' for field test_result (test_result) is invalid. Reformat to HL7 specification.",
    "errorCode" : "INVALID_MSG_PARSE_CODE"
  } ],
  "warnings" : [ ],
  "destinationCount" : 0,
  "fileName" : ""
}

emyl3 commented 1 month ago

Since deploying #7897, I'm checking what folks have submitted as codes that we don't support using the following query:

traces 
| where message has "Unsupported SNOMED result code:" 
| parse message with * "Unsupported SNOMED result code: " SNOMEDCodeEntered
| extend SNOMED = split(SNOMEDCodeEntered, ":")
| mv-expand SNOMED to typeof(string)
| summarize occurrences=count() by SNOMED

https://portal.azure.com/#@cdc.onmicrosoft.com/resource/subscriptions/7d1e3999-6577-4cd5-b296-f518e5c8e677/resourceGroups/prime-simple-report-prod/providers/Microsoft.Insights/components/prime-simple-report-prod-insights/logs

And I'm seeing, these results

CDCgov / prime-simplereport

Update Bulk Result File Validation: Result Code #7898