search

medic / cht-interoperability

Resources for enabling interoperability with the CHT based on OpenHIE and FHIR standards
GNU Affero General Public License v3.0
2 stars 3 forks source link

OpenMRS sync creates duplicate resources. #141

Open witash opened 2 weeks ago

witash commented 2 weeks ago

OpenMRS sync currently subtracts the 2 * SYNC_INTERVAL from the SYNC_PERIOD to decide how long ago to sync. So if it's syncing 1 hour ago every minute, it only syncs resources which were updated in the last 58 minutes.

This works around an issue where it was creating duplicate resources based on resources it had already created, but it does not really fix the core issues and is very confusing.

This workaround should be removed, and duplicate resources should be prevented by fixing the bug in the code that is supposed to be checking unique keys.

witash commented 2 weeks ago

The 2 * SYNC_INTERVAL from the SYNC_PERIOD is because there is a gap between when a resource is created in one system and when it is created in another system. So resources created near the boundary of SYNC_PERIOD may appear in the query for one system, but be absent from the other, causing the sync to try and create them again.

So the idea was to add a window where a resource in one system that is not in the other is not created. It's not a good solution because creation times for corresponding resources in both systems can be different by much more than 2 * SYNC_INTERVAL if the sync wasn't running or there was an error during the initial sync and it was resolved later.

witash commented 1 week ago

One fix for this could be to make sure that the FHIR server doesn't allow duplicate resources, based on id or any of the other identifiers: OpenMRS ID, CHT Document ID

So if a resource appeared in the comparison list for OpenMRS but not FHIR, but was actually in the FHIR server, the sync would still send a creation request. But it wouldn't matter, it would get a 40X error and not do anything.

njuguna-n commented 5 days ago

@witash While working on the openmrs-mediator branch the openmrs channel is not created. When checking through the logs I see this logged twice for the two mediators but the OpenMRS mediator does not have a default channel like the default mediator (see screenshots below) Image Image

I manually created the openmrs channel to move forward but then I got a 400 error when sending a patient to OpenMRS (see logs below). I have pushed some of my work to the 141-openmrs-duplicate-resources branch but I have left out a bunch of debugging logs I had added. The branch adds a defaultChannelConfig to the OpenMRS mediator, although it didn't work as mentioned above, and fixes some scripts in the startup shell script. 

2024-10-21 07:40:59 2024-10-21 04:40:59 INFO: Comparing Patient 
2024-10-21 07:40:59 2024-10-21 04:40:59 INFO: Incoming:  
2024-10-21 07:40:59 2024-10-21 04:40:59 INFO: Outgoing: d1fdf74a-0349-4133-bdee-866b14ad7b79 
2024-10-21 07:40:59 2024-10-21 04:40:59 INFO: Sending Patient d1fdf74a-0349-4133-bdee-866b14ad7b79 to OpenMRS 
2024-10-21 07:40:59 2024-10-21 04:40:59 ERROR: Request failed with status code 400 AxiosError: Request failed with status code 400
2024-10-21 07:40:59     at settle (/app/node_modules/axios/dist/node/axios.cjs:2019:12)
2024-10-21 07:40:59     at BrotliDecompress.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3135:11)
2024-10-21 07:40:59     at BrotliDecompress.emit (node:events:529:35)
2024-10-21 07:40:59     at endReadableNT (node:internal/streams/readable:1400:12)
2024-10-21 07:40:59     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2024-10-21 07:40:59     at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:4287:41)
2024-10-21 07:40:59     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

I will pick this up from the bug and fix the actual issue with the duplicates once I am back on Monday (28th of this month) and have the defaultChannelConfig automatic setup be a separate issue.

witash commented 5 days ago

setting up the defaultChannelConfig is in this branch https://github.com/medic/cht-interoperability/pull/139 the 400 is probably because of #140 . for e2e-tests the workflow adds identifier types automatically now (in https://github.com/medic/cht-interoperability/pull/139) but for manual testing, they have to be added manually (or start everything with the e2e pipeline and leave it running)

Hopefully by the time you're back we'll have these merged to openmrs_mediator