MohawkMEDIC / everest

The Everest Framework is designed to ease the creation, formatting, and transmission of HL7v3 and CDA structures with remote systems.
Apache License 2.0
38 stars 22 forks source link

Issues with parsing component.section.text with paragraph xml element #4

Closed settwi closed 7 years ago

settwi commented 7 years ago

When Everest parses certain medical section narratives within <text></text>, which are valid xml, it silently fails. For example, this section, taken from a NextGen example CDA r2:

<section>
  <templateId root="2.16.840.1.113883.10.20.22.2.12"/>
  <id root="2.201" extension="ReasonForVisit"/>
  <code code="29299-5" displayName="REASON FOR VISIT" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC"/>
  <title>Reason For Visit</title>
  <text>
    <paragraph>*** Most recent encounter only, dated '08/06/2012 10:59'. ***</paragraph>
    <paragraph>Mild Fever, 2 days</paragraph>
    <paragraph>Chills, 2 days</paragraph>
    <paragraph>Cough productive of greenish sputum, 2 days</paragraph>
  </text>
</section>

Specifically, it seems that Everest trips on the paragraph elements. I've attached a screenshot at a breakpoint showing the issue.

sc1

A potential workaround that I've thought of is simply modifying the document to replace <paragraph> and </paragraph> with <br/> before having Everest parse it, but I don't see why the framework fails on this relatively simple XML. Could it have something to do with the HL7v3 namespace being applied or not applied to the paragraph elements?

justin-fyfe commented 7 years ago

I have not experienced that behavior before however if there is a bug it is most likely in EDFormatter.Parse on DT R1 parser. Unfortunately I don’t have the cycles to investigate for the next few weeks..

On the formatter, are there any IResultDetails reported in the parse result?

Cheers -Justin

Sent from my Windows 10 phone

From: William Setterbergmailto:notifications@github.com Sent: August 26, 2017 19:03 To: MohawkMEDIC/everestmailto:everest@noreply.github.com Cc: Subscribedmailto:subscribed@noreply.github.com Subject: [MohawkMEDIC/everest] Issues with parsing component.section.text with paragraph xml element (#4)

When Everest parses certain medical section narratives within , which are valid xml, it silently fails. For example, this section, taken from a NextGen example CDA r2:

Reason For Visit *** Most recent encounter only, dated '08/06/2012 10:59'. *** Mild Fever, 2 days Chills, 2 days Cough productive of greenish sputum, 2 days

Specifically, it seems that Everest trips on the paragraph elements. I've attached a screenshot at a breakpoint showing the issue.

[sc1]https://user-images.githubusercontent.com/13972006/29742944-a54eb978-8a4d-11e7-8333-02cbdf14a739.png

A potential workaround that I've thought of is simply modifying the document to replace and with
before having Everest parse it, but I don't see why the framework fails on this relatively simple XML. Could it have something to do with the HL7v3 namespace being applied or not applied to the paragraph elements?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/MohawkMEDIC/everest/issues/4, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHqoJwrVagbOgnnJln77GVz58E0fRsGWks5scEHegaJpZM4PDiNR.

settwi commented 7 years ago

Your intuition was right. There are 16 details, all of 'em errors, but I'm not sure what to make of them. I did a bit of debug print for your convenience. Is there any way to skip validation for CV, and just pull the XML?

edit: none of the details relate to the text itself... :(

edit 2: ValidateConformance to false on the ClinicalDocumentDataTypeFormatter silences these details, but the XmlData remains null. I'll go digging into the Parse method...

also, the document in question: CCDA_Sample_NextGen_JonesIsabella.zip

Detail:
Type: Error
Exception: 
Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other'
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other'
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other'
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other'
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#code
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other'
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
Detail:
Type: Error
Exception: 
Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated
Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value
--------------------
justin-fyfe commented 7 years ago

On the formatter settings I believe you can disable validation by setting a flag to validate instances to false. I think you can also set the data types r1 formatter into CDA mode which skips quite a few validation checks.

Cheers

Sent from my Windows 10 phone

From: William Setterbergmailto:notifications@github.com Sent: August 27, 2017 02:19 To: MohawkMEDIC/everestmailto:everest@noreply.github.com Cc: Justin Fyfemailto:justin@fyfesoftware.ca; Commentmailto:comment@noreply.github.com Subject: Re: [MohawkMEDIC/everest] Issues with parsing component.section.text with paragraph xml element (#4)

Your intuition was right. There are 16 details, all of 'em errors, but I'm not sure what to make of them. I did a bit of debug print for your convenience. Is there any way to skip validation for CV, and just pull the XML?

Detail: Type: Error Exception: Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other' Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other' Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other' Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#procedure/urn:hl7-org:v3#code

Detail: Type: Error Exception: Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other' Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#code

Detail: Type: Error Exception: Message: Data type 'CV' failed basic validation, the violation was : OriginalText may only be specified when NullFlavor is not populated, or is populated with a NullFlavor that implies 'Other' Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

Detail: Type: Error Exception: Message: Data type 'CE' failed basic validation, the violation was : Translation cannot be populated unless Code is populated Locaiton: /urn:hl7-org:v3#ClinicalDocument/urn:hl7-org:v3#component/urn:hl7-org:v3#structuredBody/urn:hl7-org:v3#component/urn:hl7-org:v3#section/urn:hl7-org:v3#entry/urn:hl7-org:v3#encounter/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#act/urn:hl7-org:v3#entryRelationship/urn:hl7-org:v3#observation/urn:hl7-org:v3#value

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/MohawkMEDIC/everest/issues/4#issuecomment-325171031, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHqoJz4tw_zKao8TShIaimX-7aWiXvlGks5scMQqgaJpZM4PDiNR.

settwi commented 7 years ago

Justin, thanks for your pointers so far. I think I’ve found the issue: in the case where the XmlData of Section.Text is more than one element, ED.XmlData silently fails and returns null, as per the implementation (catch (Exception) ;) ). I’ll add another try/catch and implement a fix: I’ve found a solution using a “MultiStream” that can have beginning, data, and closing streams.

settwi commented 7 years ago

pull request 5 resolves this issue.

edit: apparently not. made a new pull request with double-fix.