SPIKE: FHIR Converter issue(s) that cause eCRs to fail to run through the pipeline

angelathe commented 2 months ago

What needs to be done

Figure out why several eCRs are failing to run through orchestration and the pipeline:

1. TN sample eCRs (3):

TN_sample eCR1: Link
TN_sample eCR5: Link
TN_sample eCR7: Link 2. KY sample eCRs (2 - all) 3. Maine sample eCRs (3 - all)
ME_sample eCR1: Link
ME_sample eCR2: Link
ME_sample eCR3: Link

See example failure messages below in Additional Info

Why it needs to be done

Describe the importance or impact of completing this task

Timebox

How much time should be dedicated to this spike?

To-do list

[x] Identify the issue(s) that cause eCRs listed above to fail to run through orchestration and the pipeline
[ ] Create any necessary tickets to fix the issue(s) and add them as blockers to the following QA tickets:
- [x] GitHub Issue: QA Test TN eCR 1
- [x] GitHub Issue: QA Test TN eCR 5
- [x] GitHub Issue: QA Test TN eCR 7
- [x] GitHub Issue: QA Test Maine eCR 1
- [x] GitHub Issue: QA Test Maine eCR 2
- [x] GitHub Issue: QA Test Maine eCR 3

Additional Info

Ex. Failure message for TN sample eCR 1:

Failed to convert TDH_1. Response: {"detail":"Service fhir_converter failed with error FHIR Converter request failed: 
{\"response\":{\"args\":\"dotnet /build/FHIR-Converter/output/Microsoft.Health.Fhir.Liquid.Converter.Tool.dll convert -- --
TemplateDirectory /build/FHIR-Converter/data/Templates/eCR --RootTemplate EICR --InputDataFile /tmp/ecr-input.txt --
OutputDataFile /tmp/output.json \",\"returncode\":255,\"stdout\":\"\",\"stderr\":\"Process failed: Error happened when 
rendering templates: Object reference not set to an instance of an object.\\n\",\"fhir_conversion_failed\":\"true\"}}"}

Ex. Failure message for TN sample eCR 5:

Failed to convert TDH_5. Response: {"detail":"Service fhir_converter failed with error FHIR Converter request failed: 
{\"response\":{\"args\":\"dotnet /build/FHIR-Converter/output/Microsoft.Health.Fhir.Liquid.Converter.Tool.dll convert -- --
TemplateDirectory /build/FHIR-Converter/data/Templates/eCR --RootTemplate EICR --InputDataFile /tmp/ecr-input.txt --
OutputDataFile /tmp/output.json \",\"returncode\":255,\"stdout\":\"\",\"stderr\":\"Process failed: Error happened when 
rendering templates: Unable to cast object of type 'System.Collections.Generic.List`1[System.Object]' to type 
'System.Collections.Generic.Dictionary`2[System.String,System.Object]'.\\n\",\"fhir_conversion_failed\":\"true\"}}"}

Ex. Failure message for TN sample eCR 7:

Failed to convert TDH_7. Response: {"detail":"Service fhir_converter failed with error FHIR Converter request failed: {\"response\":{\"args\":\"dotnet /build/FHIR-Converter/output/Microsoft.Health.Fhir.Liquid.Converter.Tool.dll convert -- --TemplateDirectory /build/FHIR-Converter/data/Templates/eCR --RootTemplate EICR --InputDataFile /tmp/ecr-input.txt --OutputDataFile /tmp/output.json \",\"returncode\":255,\"stdout\":\"\",\"stderr\":\"Process failed: Invalid JSON generated: line 73:2191 no viable alternative at input '\\\"entry\\\":[{\\\"fullUrl\\\":\\\"urn:uuid:1.2.840.114350.1.13.363.2.7.8.688883.861749262\\\",\\\"resource\\\":{\\\"resourceType\\\":\\\"Composition\\\",\\\"id\\\":\\\"1.2.840.114350.1.13.363.2.7.8.688883.861749262\\\",\\\"meta\\\":{\\\"profile\\\":[\\\"http://hl7.org/fhir/us/ecr/StructureDefinition/eicr-composition\\\"]},\\\"identifier\\\":[{\\\"use\\\":\\\"official\\\",\\\"type\\\":{\\\"coding\\\":[{\\\"code\\\":\\\"55751-2\\\",\\\"display\\\":\\\"Initial Public Health Case Report\\\",\\\"system\\\":\\\"http://loinc.org\\\",},],},\\\"value\\\":\\\"00000000-20f3-9085-0c68-e9e7bf945115\\\",\\\"assigner\\\":{\\\"display\\\":\\\"EPC\\\"},},],\\\"extension\\\":[{\\\"url\\\":\\\"http://hl7.org/fhir/StructureDefinition/composition-clinicaldocument-versionNumber\\\",\\\"valueString\\\":\\\"1\\\"},],\\\"status\\\":\\\"final\\\",\\\"type\\\":{\\\"coding\\\":[{\\\"code\\\":\\\"55751-2\\\",\\\"display\\\":\\\"Initial Public Health Case Report\\\",\\\"system\\\":\\\"http://loinc.org\\\",},],},\\\"date\\\":\\\"2024-07-30T14:34:08-05:00\\\",\\\"title\\\":\\\"Initial Public Health Case Report\\\",\\\"confidentiality\\\":\\\"N\\\",\\\"event\\\":[{\\\"period\\\":{\\\"start\\\":\\\"\\\",\\\"end\\\":\\\"\\\",},},],\\\"section\\\":[{\\\"id\\\":\\\"85fefb74-11d8-b393-7c60-f0c7bf2a88c8\\\",\\\"title\\\":\\\"Miscellaneous Notes\\\",\\\"text\\\":{\\\"status\\\":\\\"generated\\\",\\\"div\\\":\\\"<ul><li><caption>Discharge Summary - Georgette Rochelle Sevier, MD - Sat Jul 27, 2024 2:42 PM CDT</caption>Formatting of this note is different from the original. <br>7/27/2024, 2:43 PM Georgette Rochelle Sevier, MD, MDSUBMITTED BY:Pediatric Care will be at CCHS Orange Mound. Baby's Name is Idris The TardisHepatitis B vaccine DeclinedMother will make a follow up appointment for 2-4 days after discharge with CCHS , early for any problems.Discharge Follow Up: Normal newborn care, anticipatory guidance given, normal newborn handout to be given prior to discharge, discussed sleeping on back , discussed calling M.D. if rectal temperature > 100.4 F, if baby appears more jaundiced or appears dehydrated, or hearing screen and first hepatitis B vaccine prior to dischargeHomeDischarge Disposition:Discharge Cord Care: discussed cord careNewborn state screen done.Pending Labs and Tests at Discharge:Skin: Skin color, texture, turgor normal, no rashes or lesions.Neuro: Easily aroused; good symmetric tone and strength; positive root and suck; symmetric normal newborn reflexes.Extremities: Well-perfused, warm and dry.GU: Normal genitalia, FemaleHips: Negative Barlow, Ortolani, gluteal and thigh creases equal.Pulses: Strong equal femoral pulses, brisk capillary refill.Spine: Normal spine including sacrum.Abdomen: Soft, non-tender, no masses; umbilical stump clean and dry.Heart: Regular rate & rhythm, S1 S2, no murmurs, brachial and femoral pulses are equal.Chest: Lungs clear to auscultation, respirations unlabored, equal bilateral breath sounds, no tachypnea, no retractions, clavicles normal bilaterally.Neck: Supple, symmetrical, normal ROM.Throat: Lips, tongue, and mucosa are moist, pink and intact; palate intact.Nose: Clear, normal mucosa, patent nares.Ears: Well-positioned,well-formed pinnae, EAC's patent.Eyes:Sclerae white, pupils equal and reactive, red reflex normal bilaterally.Head: Sutures mobile, fontanelles normal size.General Appearance: Healthy-appearing, vigorous infant, strong cry, normal color and tone, perfused and hydrated, normal facies.Pulse 130 | Temp 98.2 °F (36.8 °C) | Resp 42 | Ht 18.5\\\"47'\\n.\\n\",\"fhir_conversion_failed\":\"true\"}}"}

angelathe commented 2 months ago

@emmastephenson Wrote the spike! Flagging for you in case you want to edit anything / put this in a specific epic

austin-hall-skylight commented 2 months ago

@emmastephenson one of the issues we are seeing comes from mapping the reasons for visit to the extension on the composition resource. Previously we were getting text or a single table. Now we have to account for one or more items, each with one or more tables, and all of these tables have different columns. I'm concerned about breaking anything that might be working currently but do we want to try to account for all the possible reasons in the different tables and concatenate them into a comma delimited string for the extension section? Maine Patient 1 eicr:

Maine Patient 2 eicr:

Working example:

output:

emmastephenson commented 2 months ago

@austin-hall-skylight good question! Normally I'd say that we should convert all possible data here. But much of this new "reason for visit" data actually looks like it's much more related to billing and insurance information, not the actual medical reason for visit. Given that, I don't know that it's super important to include it in the Viewer, esp. if those diagnoses/procedures are also present elsewhere in the record.

@sarahtress what do you think? Wondering if we need to include the fields highlighted in red in the Viewer if they're available. reason for visit

sarahtress commented 2 months ago

@austin-hall-skylight good question! Normally I'd say that we should convert all possible data here. But much of this new "reason for visit" data actually looks like it's much more related to billing and insurance information, not the actual medical reason for visit. Given that, I don't know that it's super important to include it in the Viewer, esp. if those diagnoses/procedures are also present elsewhere in the record.

@sarahtress what do you think? Wondering if we need to include the fields highlighted in red in the Viewer if they're available.

I think I need to get more caught up with what exactly is going on here bc I'm a bit confused at what I'm seeing. I agree it looks billing related, but in an important way actually. we can chat about it more in standup tomorrow?

CDCgov / phdi