search

pepfar-datim / mAtches

Data source to target mapping utility
Apache License 2.0
3 stars 1 forks source link

Updated mAtches File Format #127

Closed tomzemp closed 3 years ago

tomzemp commented 3 years ago

Hi @paynejd (FYI @Hiwot-Chichaybelu @vshioshvili @pinghuangICF @jqueipo ): I've added a sample file for the map format for a JSON file. These changes were necessitated to allow mAtches to accept JSON-formatted data.

Changes

headersStructure

This contains as Jon noted an additional key headersStructure used internally for display the tree within mAtches (https://github.com/pepfar-datim/mAtches/blob/master/public/sample_files/PLM_example_json_file_map.json#L9). This does not need to be maintained in OCL unless there is a goal to load maps from OCL in the future into mAtches.

map.headers[header].headerPath

This additional property is used to record the path within a JSON file (https://github.com/pepfar-datim/mAtches/blob/master/public/sample_files/PLM_example_json_file_map.json#L113). This is currently used within JSON conversion and hence needs to be stored in OCL, or the conversion needs to be refactored to not reference this, which would also likely be possible.

With CSV mappings, we assume that all fields are mapped and hence the map does not include unmapped headers upon submission. With a JSON structure, there are a lot of leaf nodes that we expect may not be used, so we allow for unmapped headers. Items that do not have any property except headerPath are unmapped and would not need to be persisted in OCL (provided we are not importing maps from OCL)

map.headers[header].choiceMap (this applies to maps for CSV files as well)

This change was required for supporting changes to support more flexibility for FHIR conversion (including handling answerValueSet) https://github.com/pepfar-datim/mAtches/blob/master/public/sample_files/PLM_example_json_file_map.json#L220-L224 The structure is no longer {F: Female}, but rather {F:{code: "Female", valueType: "choice"}}. This information on the valueType is currently used in conversion to FHIR and hence would be required in OCL. I could revisit to assess if this could be refactored to remove this depdendency.

paynejd commented 3 years ago

Thanks @tomzemp for laying this out so clearly -- I'll start on the following:

  1. Update QMAP import/export for the new choiceMap model
  2. Update QMAP import to ignore (rather than throw an exception) when it runs into map.headers[header].headerPath -- this will at minimum be a temporary fix so that QMAP imports will work

We can then follow up to discuss the longer-term approach to headerStructure and headerMap moving forward

paynejd commented 3 years ago

QMAP import/export code was updated to support the new choiceMap model and, as a short-term fix until we've had a chance to discuss, to ignore headerPath, headersStructure, and un-mapped CSV fields. When this is deployed to the test environment, this should allow QMAP import/export to function again without errors, though qmap exports will omit the fields mentioned above.

We plan to discuss implementing support for remaining fields on the next DASH call

paynejd commented 3 years ago

@tomzemp The QMAP import/export mediator has been updated to support unmapped CSV fields and headerPath. It has been deployed to the test.ohie server is available for integration testing.

cc @maurya

AliciaSmith876 commented 3 years ago

@vshioshvili Please confirm that this issue was resolved.

vshioshvili commented 3 years ago

Issue is resolved. We are expecting additional mAtches changes in the future that will affect OCL's code, but will address it when it comes up. Closing this for now.