num-codex / codex-processes-ap1

Business processes for the NUM CODEX project (AP1) as plugins for the HiGHmed Data Sharing Framework
Apache License 2.0
3 stars 6 forks source link

Define Conditional Update Criteria for All Resource Profiles #4

Open hhund opened 3 years ago

hhund commented 3 years ago

Conditional update criteria need to be defined for all GECCO resource profiles. For example:

https://www.netzwerk-universitaetsmedizin.de/fhir/StructureDefinition/chronic-lung-diseases

Condition?_profile=...&recorded-date=...&patient:identifier=...
hhund commented 3 years ago

The current implementation might be sufficient for most cases, but should be improved. See https://github.com/num-codex/codex-processes-ap1/blob/2037952196ac5d52fa308c35cd92f53f3eaedf27/codex-process-data-transfer/src/main/java/de/netzwerk_universitaetsmedizin/codex/processes/data_transfer/service/ReadData.java#L232-L331

storckmi commented 3 years ago

Laboratory results often contain results on several laboratory values. Therefore, these are provided with the same time stamps, since they are determined from one sample at the same time. With the given condition for the update of observation resources values would be overwritten and only the last laboratory value would be safed.

Also deleting of already sent data is difficult. If a laboratory value for a specific LOINC-code has been submitted before and is not present in the updated data it should be deleted in my opinion.

julsas commented 3 years ago

The _profile parameter might not be a good choice because the GECCO profiles do not enforce instances to populate Resource.meta.profile.

Condition: Condition?category=system|code&code=system|code&recorded-date=dateTime&subject:identifier=system|value

I started testing this with a local Hapi and Firely server. Firely server works but I'm having issues with the subject:identifier parameter on Hapi. I'll check the search parameters for other resource types next.

hhund commented 3 years ago

The send process uses a transaction Bundle as a template to search for new resources that need to be transferred to the central research repository (CRR). Since there is currently no way to transfer a pre-build transaction Bundle with our process, resources transferred to the CRR will always have a resource.meta.profile entry.

Even if we modified the search transaction Bundle to use something other then the GECCO and MII profiles to identify new or updated resource. We could always add a profile deceleration, since we would need to validate the resources not identified by a GECCO or MII profile anyways.

Currently we only validate resource while storing them in the CRR fhir-bridge, validation before Bundle generation and transmission is planed.

I have a workaround for the missing support of identifier based searches in HAPI. But since we do not store data in HAPI this is not a problem.

@julsas Is there an equivalent for Condition?category=system|code&code=system|code for other resources? Using _profile was kind of a cheap workaround to easily differentiate a lot of resources. But I agree, we should not use it for the conditional update criteria.

julsas commented 3 years ago

Procedure, Observation and MedicationStatement have .category. We currently do not use .category in MedicationStatement, but these can be differentiated bei .medicationCodeableConcept. In general the following search patterns work with a few exceptions.

DiagnosticReport: DiagnosticReport?category=system|code&code=system|code&subject:identifier=system|value&subject:identifier=system|value Problem here is that the profile currently does not enforce using a date, i.e. multiple matches possible.

Immunization: Immunization?vaccine-code=system|code&date=dateTime&target-disease=system|code&subject:identifier=system|value

MedicationStatement: MedicationStatement?status=code&code=system|code&effective=dateTime&subject:identifier=system|value

Procedure: Procedure?status=code&category=system|code&code=system|code&date=dateTime&subject:identifier=system|value The imaging procedure profile currently does not require using .category.

Observation: Observation?status=code&category=system|code&code=system|code&date=dateTime&subject:identifier=system|value Some observations currently do not require a date, like the history of travel observation.

Consent: Consent?status=code&scope=system|code&category=system|code&patient.identifier=system|value

wetret commented 3 years ago

There is another issue with the update criteria. If a date used as update criteria is not present or replaced with a data-absent-reason extension, the update criteria includes a null value and therefore leads to an error.

For example a condition with the following recordedDate

"_recordedDate":{
      "extension":[
                  {
                     "url":"http://hl7.org/fhir/StructureDefinition/data-absent-reason",
                     "valueCode":"unknown"
                  }
         ]
 }

leads to the update criteria:

Condition?_profile=https://www.netzwerk-universitaetsmedizin.de/fhir/StructureDefinition/chronic-liver-diseases&recorded-date=null&patient.identifier=http://www.netzwerk-universitaetsmedizin.de/sid/crr-pseudonym|4d8dbeb183c227fb20688e02ccaa9b112b4b466248c8fdc2b8d3c0ee1cacc12b

--> recorded-date=null