search

opensrp / opensrp-client-chw

Reference OpenSRP application for Community Health Workers (CHWs)
Other
3 stars 21 forks source link

CHW Client Processing and Sync Optimization #1018

Open rkodev opened 5 years ago

rkodev commented 5 years ago

Some users on the field are reporting extended durations during sync. These users have reporting instances where the 'Device is processing' banner is displayed for up to days at a time.

Analysis of the device's data in one such case has provided some insight:

  1. It was reported by @Marcha-star in one use case that the data was being synced offline, via p2p, and online at the same time. This use case has a probable outcome of having the device reprocess the data twice. Here are some of the solutions

a. The suggestion was to make the 2 sync protocols work exclusively. i.e syncing with the server never happens when p2p is engaged. p2p will always gracefully step down and allow server sync to complete

b. Process the p2p sync on the foreground. We suspect a lower priority thread might be killed and starved on the device. The implementation will mean during p2p will display a permanent notification on the notification bar, other suggestions welcome. This will prevent the device from sleeping

c. To keep track of all the events transferred via p2p so as not to reprocess the same event on the device. (A modification on core library)

  1. There is a probable risk that the data in the server is duplicated. In the event both the chw shares unsynced data with the supervisor, then syncs with the server, the same event might be duplicated on the server end. We have cases a few confirmed cases of duplicated data on the server. We should prioritize clean up and deploying a server fix.

  2. Upgrading the reporting library. The reporting library currently places a huge load on the DB when p2p data 'upto 4000 events in some cases' are shared. We have 2 solutions for this

a. Disabled report indicator generation completely during client processing. This relieves the load for client processing but means the Supervisor will have to wait until processing is complete before getting the device

b. We can execute these queries on demand during processing. run immediately and get live results when p2p sync is running. The DB should allow parallel access to read queries. The results will be slightly delayed but more accurate

craigappl commented 5 years ago

Thoughts: 1) If we do 1a or 1b, does that prevent 2 from happening? 2) Does the current p2p sync library already do 1c? Does the Android client know which events have not been synced, or does it just POST all events that don't have a server version to the server? 3) I think 3a should be done. I don't think it's appropriate to try to compute reports at the same time the data is being imported. I expect we should call the report to be computed after a sync process is complete. We could notify the user that the report will be updated after sync processing if it's critical.