JohnNKing commented 8 months ago

Bug

Describe the Bug

ReportStream's ORM to FHIR and FHIR to OML transformations are losing data from a variety of fields, including:

PID-3.4
PID-8.2
PID-8.3
PID-10
PID-11.7
PID-18
PID-22
NK1-2
NK1-3.3
NK1-5
NK1-33
ORC-2.2
ORC-12
OBR-2
OBR-4
OBR-7
OBR-16
OBX-3 (all except CE/CWE datatype)
OBX(2)-6 and OBX(4)-6 (NM datatypes (units))
OBX(8)-2 (XAD datatype)
OBX(8)-5 (XAD datatype)

I suspect this was introduced by: https://github.com/CDCgov/prime-reportstream/pull/12718

For more details, see https://slack.com/archives/C048S3GPSQG/p1707861774688789

Also ensure an error isn't caused when the order has no OBX segments. (i.e. make OBX not required for ORM messages). Potentially relevant: https://github.com/CDCgov/prime-reportstream/blob/0eb5b42c5a8ccb2d7876fbee7441c790eaf5b828/prime-router/src/main/resources/metadata/hl7_mapping/OML_O21/OML_O21-base.yml#L62

Recently transformed ORM (from 9/11/2024): https://www.slack.com/archives/C044P2CBEKS/p1726088886147019?thread_ts=1726088335.991899&cid=C044P2CBEKS

Impact

ORM orders are lacking key pieces of information for NBS.

To Reproduce

Generate a JWT using the Flexion staging API key for ReportStream
Use the ReportStream login endpoint to get a session JWT
Send the following to the Water's endpoint: https://github.com/CDCgov/trusted-intermediary/blob/main/examples/MN/003_MN_ORM_O01_NBS.hl7
5 min later, use the RS admin UI's daily-data page to download the resulting HL7 file

Expected Behavior

Expected HL7 output should be the following (or equivalent):

MSH|^~\&|Epic^1.2.840.114350.1.13.145.2.7.2.695071^ISO|Centracare^centracare.com^DNS|NATUS^natus.health.state.mn.us^DNS|MN Public Health Lab^2.16.840.1.114222.4.1.10080^ISO|20230506052916-0500||OML^O21^OML_O21|31808297|P|2.5.1|||AL|AL|||||LAB_PRU_COMPONENT^^2.16.840.1.113883.9.82^ISO~LAB_TO_COMPONENT^^2.16.840.1.113883.9.22^ISO
PID|1||11102779^^^^MR||SMITH^BB SARAH^^^^^L|SMITH^SADIE^S|20230504131023-0500|M|||555 STATE HIGHWAY 13^^DEER CREEK^IG^56527-9657^USA^H^^OTTER TAIL||^^PH^^^763^555555^^(763)555-5555|||||||||||N||||||N
NK1|1|SMITH|MTH^mother|555 STATE HIGHWAY 13^^DEER CREEK^IG^56527-9657^USA^H^^OTTER TAIL|^PRN^PH
ORC|NW|421832901^EPIC|||unknown||||20230506102913+0000
OBR|1|||54089-8^Newborn screening panel American Health Information Community (AHIC)^LN|||20230506050000+0000||||P
OBX|1|NM|8339-4^BIRTH WEIGHT MEASURED^LN|1|1769.8593|g^gram^UCUM|||||unknown|||20230506050000-0500
OBX|2|NM|57714-8^OBSTETRIC ESTIMATION OF GESTATIONAL AGE^LN|1|32|wk^week^UCUM|||||unknown|||20230506050000-0500
OBX|3|CWE|57713-0^INFANT FACTORS THAT AFFECT NEWBORN SCREENING INTERPRETATION^LN|1|LA12419-0^INFANT IN NICU AT TIME OF SPECIMEN COLLECTION^LN||||||unknown|||20230506050000-0500
OBX|4|CWE|MNDSIB^DOES THE BABY HAVE A DECEASED SIBLING?^99MDH|1|N^No^99MDH||||||unknown|||20230506050000-0500
OBX|5|CWE|MNDEFECT^DOES THE BABY HAVE BIRTH DEFECTS?^99MDH|1|N^No^99MDH||||||unknown|||20230506050000-0500
OBX|6|CWE|MNFAM^FAMILY HISTORY OF DISORDER ON MN SCREENING PANEL^99MDH|1|N^No^99MDH||||||unknown|||20230506050000-0500
OBX|7|CWE|67704-7^FEEDING TYPES^LN|1|LA12418-2^TPN^LN||||||unknown|||20230506050000-0500
OBX|8|CWE|67704-7^FEEDING TYPES^LN|2|LA16914-6^BREAST MILK^LN||||||unknown|||20230506050000-0500
OBX|9|CWE|57723-9^UNIQUE BAR CODE NUMBER OF CURRENT SAMPLE^LN|1|0516199364||||||unknown|||20230506050000-0500
OBX|10|ST|62328-0^DISCHARGE PROVIDER PRACTICE PHONE NUMBER^LN|1|Daniel Davis/218-555-1000||||||unknown|||20230506050000-0500
OBX|11|ST|62324-9^POST-DISCHARGE PROVIDER NAME^LN|1|DAVIS, DANIEL, MD||||||unknown|||20230506050000-0500
OBX|12|ST|62326-4^POST-DISCHARGE PROVIDER PRACTICE NAME^LN|1|Lakeridge Health Revere||||||unknown|||20230506050000-0500
SPM|1|||440500007^Blood spot specimen^SCT|||||||||||||20230506050000

Tasks

[x] Reach out to ReportStream to better understand the current status
[x] Test some of our sample ORMs against the ReportStream PR: https://github.com/CDCgov/prime-reportstream/pull/15910
[x] Review tests that are part of this PR; determine if those tests covers all of our data needs (or not)
[x] Provide feedback to RS team if we find any gaps ~- [ ] Look into possibility of adding assertions in the RS Integration Test framework to catch mapping issues~
[x] Remove OBR-11 transformation from etor-oml-receiver-transform.yml

Screenshots

n/a

Logs

n/a

Version

Current master branch

Additional Context

n/a

eaugustini commented 8 months ago

While reviewing AL test order scenarios output, I found that these fields were missing in the output OML PID-3.4 PID-8.2 PID-8.3 PID-10 PID-11.7 PID-18 PID-22 NK1-2 NK1-3.3 NK1-5 NK1-33 ORC-2.2 ORC-12 OBR-2 OBR-4 OBR-7 OBR-16 OBX-3 (all except CE/CWE datatype) OBX(2)-6 and OBX(4)-6 (NM datatypes (units)) OBX(8)-2 (XAD datatype) OBX(8)-5 (XAD datatype)

JohnNKing commented 1 month ago

Original list of missing fields: - OBX-3

OBX-6
OBX-5-3
OBR-4
OBR-7
ORC-21
SPM-4
SPM-17

JohnNKing commented 1 month ago

Existing ReportStream epic: https://app.zenhub.com/workspaces/platform-6182b02547c1130010f459db/issues/gh/cdcgov/prime-reportstream/14958

PR: https://github.com/CDCgov/prime-reportstream/pull/1591

basiliskus commented 1 month ago

PR in RS: https://github.com/CDCgov/prime-reportstream/pull/15910. The PR has been merged into this branch: OML-Updates-Base

jbiskie commented 1 month ago

This is the E2E output of the sample message linked in the summary when run against the OML-Updates-Base feature branch post-PR #15910 merge.

MSH|^~\&|Epic^1.2.840.114350.1.13.145.2.7.2.695071^ISO|Centracare^centracare.com^DNS|NATUS^natus.health.state.mn.us^DNS|MN Public Health Lab^2.16.840.1.114222.4.1.10080^ISO|20230506052916-0500||OML^O21^OML_O21|31808297|P|2.5.1|||AL|AL|||||LAB_PRU_COMPONENT^^2.16.840.1.113883.9.82^ISO~LAB_TO_COMPONENT^^2.16.840.1.113883.9.22^ISO
PID|1||11102779^^^CRPMRN^MR||SMITH^BB SARAH^^^^^L|SMITH^SADIE^S|20230504131023-0500|M|||555 STATE HIGHWAY 13^^DEER CREEK^MN^56527-9657^USA^H^^OTTER TAIL||^^PH^^^763^5555555^^(763)555-5555|||||||||||N|1|||||N
ORC|NW|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO|||||||20230506052913-0500|||1265136360^JONES^JANE^^^^^^NPI^L^^^NPI|||||||||ST. CLOUD HOSPITAL^^^^^CMS^NPI^^^1043269798~ST. CLOUD HOSPITAL^^^^^MN Public Health Lab^Submitter ID^^^739
OBR|1|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO|||||||||P

jbiskie commented 3 weeks ago

ReportStream HL7 to FHIR output of https://github.com/CDCgov/trusted-intermediary/tree/main/examples/MN/003_MN_ORM_O01_NBS.hl7 when run on the OML-Updates-Base feature branch:

LA ORM converted to FHIR on OML-Updates-Branch.json

basiliskus commented 1 week ago

I submitted 003_MN_ORM_O01_NBS.hl7 locally using the OML-Updates-Base branch in RS which has the new ORM/OML mappings. It looks like this branch fixed most of the issues we were seeing. For 003_MN_ORM_O01_NBS.hl7, the only difference is that one of the OBRs is removed in the final HL7 and OBR-7 and OBR-15 are missing for the one preserved. Here's the content:

MSH|^~\&|Epic^1.2.840.114350.1.13.145.2.7.2.695071^ISO|Centracare^centracare.com^DNS|NATUS^natus.health.state.mn.us^DNS|MN Public Health Lab^2.16.840.1.114222.4.1.10080^ISO|20230506052916-0500||OML^O21^OML_O21|31808297|D|2.5.1|||AL|AL|||||LAB_PRU_COMPONENT^^2.16.840.1.113883.9.82^ISO~LAB_TO_COMPONENT^^2.16.840.1.113883.9.22^ISO
PID|1||11102779^^^CRPMRN^MR||SMITH^BB SARAH^^^^^L|SMITH^SADIE^S|20230504131023-0500|M|||555 STATE HIGHWAY 13^^DEER CREEK^MN^56527-9657^USA^H^^OTTER TAIL||^^PH^^^763^5555555^^(763)555-5555|||||||||||N|1|||||N
ORC|NW|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO|||||||20230506052913-0500|||1265136360^JONES^JANE^^^^^^NPI^L^^^NPI|||||||||ST. CLOUD HOSPITAL^L^^^^CMS^NPI^^^1043269798~ST. CLOUD HOSPITAL^L^^^^MN Public Health Lab^Submitter ID^^^739
OBR|1|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO||54089-8^Newborn screening panel American Health Information Community (AHIC)^LN|||||||P
OBX|1|NM|8339-4^BIRTH WEIGHT MEASURED^LN|1|1769.859285|g^gram^UCUM|||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|2|NM|57714-8^OBSTETRIC ESTIMATION OF GESTATIONAL AGE^LN|1|32|wk^week^UCUM|||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|3|CWE|57713-0^INFANT FACTORS THAT AFFECT NEWBORN SCREENING INTERPRETATION^LN|1|LA12419-0^INFANT IN NICU AT TIME OF SPECIMEN COLLECTION^LN||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|4|CWE|MNDSIB^DOES THE BABY HAVE A DECEASED SIBLING?^99MDH|1|N^No^99MDH||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|5|CWE|MNDEFECT^DOES THE BABY HAVE BIRTH DEFECTS?^99MDH|1|N^No^99MDH||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|6|CWE|MNFAM^FAMILY HISTORY OF DISORDER ON MN SCREENING PANEL^99MDH|1|N^No^99MDH||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|7|CWE|67704-7^FEEDING TYPES^LN|1|LA12418-2^TPN^LN||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|8|CWE|67704-7^FEEDING TYPES^LN|2|LA16914-6^BREAST MILK^LN||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|9|CWE|57723-9^UNIQUE BAR CODE NUMBER OF CURRENT SAMPLE^LN|1|0516199364||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|10|ST|62328-0^DISCHARGE PROVIDER PRACTICE PHONE NUMBER^LN|1|Daniel Davis/218-555-1000||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|11|ST|62324-9^POST-DISCHARGE PROVIDER NAME^LN|1|DAVIS, DANIEL, MD||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|12|ST|62326-4^POST-DISCHARGE PROVIDER PRACTICE NAME^LN|1|Lakeridge Health Revere||||||O|||20230506050000-0500|||||||||||||||QST|AOE

GilmoreA6 commented 1 week ago

The 2nd OBR is being dropped because 003_MN_ORM_O01_NBS.hl7 isn't a valid ORM per HL7 V2.5.1. 2.5.1 specifies that the ORC segment is required for each order group, adding a second ORC above the second OBR produces both OBRs:

MSH|^~\&|Epic^1.2.840.114350.1.13.145.2.7.2.695071^ISO|Centracare^centracare.com^DNS|NATUS^natus.health.state.mn.us^DNS|MN Public Health Lab^2.16.840.1.114222.4.1.10080^ISO|20230506052916-0500||ORM^O01^ORM_O01|31808297|D|2.5.1|||AL|AL|||||LAB_PRU_COMPONENT^^2.16.840.1.113883.9.82^ISO~LAB_TO_COMPONENT^^2.16.840.1.113883.9.22^ISO
PID|1||11102779^^^CRPMRN^MR||SMITH^BB SARAH^^^^^L|SMITH^SADIE^S|20230504131023-0500|M|||555 STATE HIGHWAY 13^^DEER CREEK^MN^56527-9657^USA^H^^OTTER TAIL||^^PH^^^763^5555555^^(763)555-5555|||||||||||N|1|||||N
ORC|NW|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO|||||||20230506052913-0500|||1265136360^JONES^JANE^^^^^^NPI^L^^^NPI|||||||||ST. CLOUD HOSPITAL^L^^^^CMS^NPI^^^1043269798~ST. CLOUD HOSPITAL^L^^^^MN Public Health Lab^Submitter ID^^^739
OBR|1|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO||54089-8^Newborn screening panel American Health Information Community (AHIC)^LN|||||||P
ORC|NW|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO|||||||20230506052913-0500|||1265136360^JONES^JANE^^^^^^NPI^L^^^NPI|||||||||ST. CLOUD HOSPITAL^L^^^^CMS^NPI^^^1043269798~ST. CLOUD HOSPITAL^L^^^^MN Public Health Lab^Submitter ID^^^739
OBR|2|421832901^EPIC^1.2.840.114350.1.13.145.2.7.2.695071^ISO||57717-1^Newborn screen card data panel^LN
OBX|1|NM|8339-4^BIRTH WEIGHT MEASURED^LN|1|1769.859285|g^gram^UCUM|||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|2|NM|57714-8^OBSTETRIC ESTIMATION OF GESTATIONAL AGE^LN|1|32|wk^week^UCUM|||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|3|CWE|57713-0^INFANT FACTORS THAT AFFECT NEWBORN SCREENING INTERPRETATION^LN|1|LA12419-0^INFANT IN NICU AT TIME OF SPECIMEN COLLECTION^LN||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|4|CWE|MNDSIB^DOES THE BABY HAVE A DECEASED SIBLING?^99MDH|1|N^No^99MDH||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|5|CWE|MNDEFECT^DOES THE BABY HAVE BIRTH DEFECTS?^99MDH|1|N^No^99MDH||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|6|CWE|MNFAM^FAMILY HISTORY OF DISORDER ON MN SCREENING PANEL^99MDH|1|N^No^99MDH||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|7|CWE|67704-7^FEEDING TYPES^LN|1|LA12418-2^TPN^LN||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|8|CWE|67704-7^FEEDING TYPES^LN|2|LA16914-6^BREAST MILK^LN||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|9|CWE|57723-9^UNIQUE BAR CODE NUMBER OF CURRENT SAMPLE^LN|1|0516199364||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|10|ST|62328-0^DISCHARGE PROVIDER PRACTICE PHONE NUMBER^LN|1|Daniel Davis/218-555-1000||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|11|ST|62324-9^POST-DISCHARGE PROVIDER NAME^LN|1|DAVIS, DANIEL, MD||||||O|||20230506050000-0500|||||||||||||||QST|AOE
OBX|12|ST|62326-4^POST-DISCHARGE PROVIDER PRACTICE NAME^LN|1|Lakeridge Health Revere||||||O|||20230506050000-0500|||||||||||||||QST|AOE

basiliskus commented 1 week ago

Submitting the LA example examples/LA/001_LA_ORM_O01_NBS.hl7, the final HL7 is:

Missing both NK1 segments
Missing OBR-7 and OBR-15
Added P value in OBR-11

GilmoreA6 commented 1 week ago

additional information regarding above:

the NK1 segments are missing in the output because NK1 is not a valid segment in an ORM-001 message but is a valid segment in an OML-021. The segments are getting lost because RS has no mapping for NK1 to the "RelatedPerson" FHIR resource for an ORM message. If the NK1 segment needs to be passed for LA, we would need to create separate mappings for this or have that information put into a valid segment for an ORM message.
The "P" is explicitly being added in receiver transform "etor-oml-receiver-transform.yml". This can be easily changed.
OBR-7 is not being populated because the FHIR-> HL7 mappings for an OML_021 expects this tp be populated with information from the FHIR "Specimen" resource which is not created because ORM messages do not contain an SPM segment (I have reached out to RS to make them aware of this as this is bug since that resource should still be created as information from the OBR populates into it).
OBR-15 is not being populated because 2.5.1 OML-021 messages only have the OBR-15 field for backwards compatibility so there is not mapping for this field.

brick-green commented 6 days ago

@GilmoreA6 Is going to make a new ticket for the NK1 work listed above.

What is the plan for receiver transform "etor-oml-receiver-transform.yml" changes? Is that needed to close out this ticket?

basiliskus commented 6 days ago

@brick-green good point. We need to figure out if we need that transformation for OBR-11

basiliskus commented 6 days ago

@brick-green Confirmed that we should remove that transformation, so that task should be part of this ticket

brick-green commented 6 days ago

@basiliskus @GilmoreA6 Can you add that task above? Can you also review the other tasks and check off the ones that are complete?

basiliskus commented 6 days ago

@brick-green added it

GilmoreA6 commented 6 days ago

@brick-green there is a task "Look into possibility of adding assertions in the RS Integration Test framework to catch mapping issues" to me that also seems like a separate issue, since this ticket is specifically about validating the RS mappings and reporting gaps, thoughts?

brick-green commented 5 days ago

The integration testing will be broken out into a new ticket(s). Remaining work is removing the transformation file.

basiliskus commented 5 days ago

Added this PR in RS to remove the etor-oml-receiver-transform.yml transform

JohnNKing commented 5 days ago

Discussed in Stream 2 Refinement in relation to #472; we're assuming at this point that SPM will be provided, which would provide an alternate means of getting the info from OBR-15.

basiliskus commented 4 days ago

PR to remove etor-oml-receiver-transform.yml in RS has been merged

CDCgov / trusted-intermediary

ReportStream ORM/OML Transformations Losing Data #900

Bug

Describe the Bug

Impact

To Reproduce

Expected Behavior

Tasks

Screenshots

Logs

Version

Additional Context