search

nmrs-nigeria / openmrs-module-nigeriaemr

NMRS NDR Extration Module for NigeriaMRS
1 stars 15 forks source link

"Timestamping" the container object to detect deleted encounter #141

Closed brightoibe closed 3 years ago

brightoibe commented 4 years ago

Am proposing a new attribute called "change_timestamp" to be added to the container of every patient's xml. NDR maintains a timstamp history of all patients. If the timestamp on your XML is different from the timestamp on NDR, NDR will consume all your records as "INITIAL". But if the timestamp on NDR is the same as the timestamp on the container, it means no change has happend to your record, NDR has no need to even parse your XML. This will improve the speed and efficiency of the NDR as NDR focuses on consuming only "the entire records" of patients where a new encounter has been added or modified.

wandechris commented 4 years ago

@brightoibe is this not the purpose of INITIAL and UPDATE flag on the current export, if the timestamp is the same, then the patient XML shouldn't have left the emr in the first place

brightoibe commented 4 years ago

This is to completely eliminate the need to store anything in the global properties. The process will be completly automated. All XML files are exported at every export but only the files with a "change in their timestamp" are consumed by NDR.

brightoibe commented 4 years ago

Scenerio 1 : 10,000 XML files are exported by the plugin, NDR checks the timestamp property in the container of each XML file before consuming the file, if your timestamp has changed from what it was the last time, your file is elibile for consumption, if your timestamp is still the same from what it was the last time then your file is not eligible for ingestion.

brightoibe commented 4 years ago

The plugin scans all the tables where a change is likely to occur e.g (patient, person, person_address, patient_program, obs, encounter,etc) and gets the maximum date_created or date_modified across all these tables to create a variable called "change_timestamp" which is stored as the attribute of the container.

brightoibe commented 4 years ago

This implication of this is that NDR will do a full ingestion of any XML file that is eligible for ingestion discarding any previous record that patient had in the past.

brightoibe commented 4 years ago

This will solve the problem of constant requests for sites to be droped on NDR and get us to achieveing 100% concurrence between EMR and NDR

wandechris commented 4 years ago

this approach will bring new problems

  1. each export will take a very long time to complete because all patient data will be exported.
  2. why export 1000 patient data when there's a chance none of them is needed.
  3. we'll be transferring the task of selecting the only patient that changes to the NDR when it's easier done on the NMRS level.

the solution is for the NDR_last_run_date to not be modified by anyone.

I understand why this is being done today, but we've discussed and will be eliminating the need for that.