Syncing requirements

[mberg]

When evaluating potential new approaches for syncing. Some of the requirements we will want to support.

• Ability to create and update local entities on the phone
• Ability to sync when there is connectivity
• Ability to sync a subset of records for a patient or client. We don't want to force downloading of entire form history of a health worker.
• Ability to download patients from another health worker. This would be similar to syncing a cohort
• Ideally support peer to peer syncing. Ie) update a record on another health workers phone if it's been updated

[mberg]

Explore this:

http://developer.couchbase.com/mobile/develop/guides/sync-gateway/

[mberg]

https://cloudant.com/blog/introducing-cloudant-sync/#.VbcXceiqqko

[mberg]

https://github.com/cloudant/sync-android

[ndisha]

Some of the suggestions that Matt passed on today and the possible fixes discussed:

1. @maimoonak suggested we use locationID to download data rather than using the anmID
2. @mberg we sync the database details (just what changed) rather than the whole form submission. This means the ziggy will remain on the client side to map the form submissions to the form definitions, in summary we will:

-use ziggy for entity mapping relationship -we can explore merging the mother, child and ec tables (they have similar details) -we are exploring synching of databases -this will lead to changing the ziggy architecture on client for the syncing part

We can we have a history of activities that happened for this client, would these be like a report of sorts? Where do we view these, at the client side or the server side? We explore an approach where we use couch syncing to update a master table and use filters to figure out data the user has access to. Check the email that @koros sent and keep the conversation going.

@sohelsarder @raihan-mpower @euclidian @dimasciput @cagulas Can we put our concerns here?

[sohelsarder]

1. Data should have download option by locationID, by anmID or by both that would add more download flexibilities.

2. In concern to Ziggy architecture changes, So far my understanding is, ziggy is doing:

   1. Entity mapping (according to form definition.json )
      2. Maintaining Entity relationship (according to entity relationship json)

   which in other words generally dictating which part of submission going into what part of the database's which particular field. Basically ziggy at client side parsing and loading enketo collections into sqllite tables and maintaining form submission sync with opensrp server. And at sever side doing almost the same thing but this time it is parsing and loading form submission and into couchDB. So from very generic view it is simply doing entity class properties value mapping and data parsing that we can easily replace with java class instead of taking help from javascript which is sometimes difficult to understand and change for the developers. And it would be great if we could get rid of that mysterious part. But before start of that implementation we need to ensure the following things, at least a stable version of code at repository so that other project can fork over it and can continue its ongoing implementation before ziggy replacement done and replacement should be done in a way so that these forked version can easily converge with main stream without huge migration effort. We need to keep in mind the other priorities of the platform that we also need to implement like,

3. Schedule data passing into openmrs (ensuring no data transmission loss between opensrp to openmrs).
4. Robustness of opensrp to openmrs transmission error handling and error reporting so that opensrp developers can integrate openmrs with minimal understanding of openmrs internals can troubleshoot the error from opensrp application scope.
5. Multimedia data transmission server part not ready yet.
6. Enrich opensrp api module for easy integration with other external systems.
7. Finding other smart alternatives of data syncing (i.e check the prospect of cloud sync)

so far these come to my mind others ( @maimoonak @raihan-mpower @euclidian @dimasciput @cagulas @mberg ) can raise their part as well i think we need a balance decision here so that it minimally impact its dependent project development milestone and other platform specific priorities as well.

[maimoonak]

@mberg @sohelsarder @ndisha @alihabib @raihan-mpower @dimasciput @djazayeri

\ Although the story below looks uselessly long and boring but please read it thoroughly and provide your valuable feedback because it is related to core and base of OpenSRP. **

The expected syncing requirements seem massive refactoring and complete change of process and data flow on client side and client developers can comment on this better. Since changes are massive these can not be done immediately and would need smooth transition that should not affect currently running app (or in worst case minimal impact). Below is my analysis (concerned with server side work) and suggestions on how to transit.

1 - Current Flow:

All data is submitted and synced as FormSubmission on server/mobile-client that is parsed into custom register entities via Ziggy.

Problem: Current server register module doesnot enroll the beneficiary and its details into a central master entity (with a static and standard predefined structure) that can be verified and queried from any internal or external app without detailed knowledge of custom entities, register architecture and form submission fields structure.

Solution:

• We have master tables already designed with static and predefined architecture conforming to OpenMRS and any other application that is unlikely to change unless a new feature is added to OpenSRP and the documentation written for this would be easier to follow, maintain and to extend the functionality would be easier.
• This architecture is Client, Events and Obs which can be queried with any filters (may need slight refactoring and functionality building in core API). Leaving this to register developer would be useless as we can expect that developers would intentionally or unintentionally deviate the protocol if they are given the choice. Rather, before parsing the entities on server on new FormSubmission(done in core dynamically) the system would automatically create Client, Event and Obs from FormSubmission and then would route the data to Handler implemented by register developer as it is rightnow. This is easy and would not require massive refactoring on any side of code.
• The new data would be submitted as FormSubmissions

2 - Current Flow:

Every device to server sync includes download of all non-existing FormSubmissions on device for that HealthWorker from server. If this is first login or new device it leads to unlimited download with each and every data that may never be needed on mobile-client

Problem: There are three different potential problems in this approach 1- Bulk data download from start of program to date with every minute detail 2- The data on download goes under form files loading and reading, JSON parsing, table creations and data push in multiple tables i.e. avoidable heavy IO operations. The approach is inherent into app and trying changing the process would lead to drastic changes in mobile-client code 3- HealthWorker has access to his/her beneficiaries only

Solution: 1- As @mberg suggested that the approach would be changed and in future mobile-client would sync data directly from server (sqlite to couch). This would be supported by the change suggested in point 1 where we would have all data as Client, and Event and Obs (standard, central unlikely to change model). This would allow mobile-client send a filtered request for data download. The filter could be

• all FormSubmissions for forms specified (do we need it)
• all FormSubmissions for HealthWorker specified HealthWorkers in specified date range (do we need it)
• all Clients only
• all Clients filtered by event types
• all Clients registered in specified date range
• all Events filtered by event types
• all Events in specified date range
• all FormSubmission or Events of specified Location (do we need it)
• Events would optionally have Obs data

2- The change would lead to massive refactoring on mobile-client and @raihan-mpower and @dimasciput can better comment on it, but from server side all above filters would be available and mobile-client can then go into this transition smoothly. New FormSubmissions on mobile-client would submit data to server as it is rightnow. Server would create Client and Events out of this and any other fresh device can download data as filtered Client and Event or even FormSubmissions.

3- The problem would be solved by using the approach above as HealthWorker would be able to download data of his/her own choice using different filters.

• HealthWorker would also have a search and claim Client feature where he/she would able to search Clients/Events based on identifiers, some attributes and locations (@raihan-mpower and @dimasciput).
• Data privacy needs to be discussed on how much details to be allowed sent to mobile-client based on HealthWorker Role and Location.

This would also allow external apps to sync and query data by following a standard, simple and unlikely to change Documentation.

Suggested Change in point 2

The changes suggested in syncing are: direct syncing between databases (sqlite to couch) with some master entities (Client and Events n Obs) incase of existing data sync or incase of edited data sync. The FormSubmissions would only be used to submit new Forms.

Problem: 1- Massive change on mobile-client and server 2- Direct sync by passing application logic

• Direct syncing would not generate any system event that can alert application to push data to OpenMRS.
• Direct push of data from mobile-client to OpenMRS is not a good idea as it would make app tightly coupled with OpenMRS and also end user app (Views and UI related components of system) managing business logic is a breach of OO principles.
• Data privacy and security would need a detailed think through; a mobile-client via HTTP sending data to override server DB !!!.
• Validations of data would not be possible to implement.

3- The sync still included full syncing of Client and/or Event data

• Although sync would be narrowed down but incase of editing of data we would still need to sync full Client and/or Event record. The idea of managing all details in a single master table would not work as data has to be designed and normalized well (or maybe I misinterpreted the statement).

4- Inconsistency in FormSubmission and Client and Event and Obs incase of edits

• It would lead to an inconsistency in FormSubmission (submitted when filling a form) and Client and Event data. Rightnow FormSubmission is used to manage data edits via serverVersion i.e. timestamp of edit.

Solution: 1- (@raihan-mpower and @dimasciput) any concerns or suggestions on mobile-client changes !?

• Server would also undergo massive refactoring but its doable as Ziggy part is going to be limited to determine entity relationships only.

2- Direct syncing of databases would lead to a lot of code wasted though. OpenMRS-connector would also need refactoring.

3- @mberg and team !!?

4- Incase of data edits these would be done and synced via Client and Event n Obs, instead of FormSubmissions.

• Client and Event already have audit information so finding modified records is not a big deal on any side. But considering FormSubmission inconsistency is required.
• This would also make peer-peer syncing possible.

Concerns and Answers by each point raised by team.

• Ability to create and update local entities on the phone

    - @raihan-mpower and @dimasciput : OpenSRP API has this flexibility but mobile-client 
team can suggest how to use that

• Ability to sync when there is connectivity

    - It is part of system rightnow and would remain as it is i.e. repeated auto data sync if 
network is available. Client app can allow user to filter and configure from 
settings/preferences
    - Direct sync would also be configurable for sure (@mberg)

• Ability to sync a subset of records for a patient or client. We don't want to force downloading of entire form history of a health worker.

    - see points discussed above.

• Ability to download patients from another health worker. This would be similar to syncing a cohort

    - points discussed above solve the problem.
    - "similar to syncing a cohort" needs more explanation

• Ideally support peer to peer syncing. Ie) update a record on another health workers phone if it's been updated

    - Client and Event has audit data and whenever record is updated it could be sent to 
mobile-client dealing with those beneficiaries (by location/HealthWorker or anything else). 
Ex: System on syncing request mobile-client finds the updated Client/Event applicable for 
that HealthWorker from  last sync to date and sends it to calling device.
    - Important things to consider
        - How to edit data on mobile-client (rightnow its FormSubmission)
        - Sometimes editing should be another FormSubmission. i.e. Closing an 
ANC case is an event in the life of beneficiary. It should not be handled via edit.
        - A person`s marital status should never be edited directly (unless data entry 
was wrong). It is an important event in life of person that may require some other 
details and modifications. E.g. Women would be moved out from a household.

• We sync the database details (just what changed) rather than the whole form submission. This means the Ziggy will remain on the client side to map the form submissions to the form definitions, in summary we will:

    - see points above

-use ziggy for entity mapping relationship

    - data saving is embedded into script and changing it would need refactoring 
Ziggy javascript file (although a mix of java and javascript code).

-we can explore merging the mother, child and ec tables (they have similar details)

    - Client and Event n Obs solves the problem.

-we are exploring synching of databases

    - see point 2 and its concerns to take into account

-this will lead to changing the ziggy architecture on client for the syncing part

    - if team working on syncing, share the proposed plan with us, we might be 
able to suggest or share our concerns better

• We can we have a history of activities that happened for this client, would these be like a report of sorts? Where do we view these, at the client side or the server side?

    - I would suggest to use Client and Event n Obs and the audit data available to manage this.
 If there is any other thing team has in mind plz share with us. Also I would recommend to 
stick to a standard and single model for everything rather than using different types of 
models for different purposes. This also leaves potential inconsistencies in data.

• We explore an approach where we use couch syncing to update a master table and use filters to figure out data the user has access to.

    - Does Client and Event n Obs fits it?

• Data should have download option by locationID, by anmID or by both that would add more download flexibilities.

    - addressed in point 1 and 2

• In concern to Ziggy architecture changes, So far my understanding is, ziggy is doing:

  1. Entity mapping (according to form definition.json )
  2. Maintaining Entity relationship (according to entity relationship json)

  which in other words generally dictating which part of submission going into what part of the database's which particular field. Basically ziggy at client side parsing and loading enketo collections into sqllite tables and maintaining form submission sync with opensrp server. And at sever side doing almost the same thing but this time it is parsing and loading form submission and into couchDB. So from very generic view it is simply doing entity class properties value mapping and data parsing that we can easily replace with java class instead of taking help from javascript which is sometimes difficult to understand and change for the developers. And it would be great if we could get rid of that mysterious part. But before start of that implementation we need to ensure the following things, at least a stable version of code at repository so that other project can fork over it and can continue its ongoing implementation before ziggy replacement done and replacement should be done in a way so that these forked version can easily converge with main stream without huge migration effort. We need to keep in mind the other priorities of the platform that we also need to implement like, Schedule data passing into openmrs (ensuring no data transmission loss between opensrp to openmrs). Robustness of opensrp to openmrs transmission error handling and error reporting so that opensrp developers can integrate openmrs with minimal understanding of openmrs internals can troubleshoot the error from opensrp application scope. Multimedia data transmission server part not ready yet. Enrich opensrp api module for easy integration with other external systems. Finding other smart alternatives of data syncing (i.e check the prospect of cloud sync)

    - Ziggy doesnot sync with server. It parses the FormSubmissions received from server into 
custom entities like ec, mother and child and saves both FormSubmissions and custom 
entities into sqlite. This is done by Ziggy but controlled by java classes 
(Ziggy service is passive).
    - On server it doesnot load and save FormSubmission rather it get FormSubmissions to be
 processed(controlled by Active java service) and parses into entities and then saves those 
in their corresponding tables.
    - If data sync is done sqlite-couch then we would get Ziggy removed from server, so 
rightnow the priority is to refactor and minimize its role on server rather than replicating 
the functionality.
    - Although I don`t feel like data replication for scheduling on OpenMRS is a good idea 
because scheduling generates daily alerts for HealthWorker and its state is maintained in 
two or three different entities (Action, scheduleTrackingApi internal domain model). We 
can think of pushing data to server when state goes into state Completed/Defaulted 
otherwise it would be back and forth updates on both sides. But still it depends on 
what reports we would ever need from scheduling API. Please donot discuss this point 
in this thread. Copy and paste it in different ticket and provide your comments/feedback
    - Multimedia data transmission: Is anyone working on this from your side?
    - Easy integration with other external systems with opensrp: Please list down 
in detail, the current problems, your requirements and suggestions in this regard 
in a separate ticket
    - For data syncing provide your feedback, suggestions and concerns on approach 
point 1 and 2

[koros]

@mberg @sohelsarder @ndisha @alihabib @raihan-mpower @dimasciput @djazayeri @maimoonak We can put this feature under R&D, as pointed out there are some considerations and challenges to be addressed, nonetheless I belief we should continue exploring this issue as well as other suggested options, we can always refer back to it in future if need be. It's also expected other obstacles could emerge down the line, for example the initial bootstrapping of the data if we are doing table to table synching the only table that exist on the server side at the moment is FormSubmissions table, however if in the end the pros outweigh the cons we can see how to resolve the challenges, e.g we could use a batch job to process and submit the synced data

[ndisha]

@maimoonak, thanks for this; @raihan-mpower @koros @dimasciput I have taken point 2 and added my comments on the form submission data optimization on the openrsp client side: https://github.com/OpenSRP/opensrp-client/issues/8;

[julkarnain]

@maimoonak @ndisha @sohelsarder I am explaining the understanding of ziggy in addition to Maimoona's concept. Ziggy converts the raw submission data to the simple and splited format depends on the bind type. Ziggy iterate the form submission data recursively. When it works with every bind type for form and sub-form both.

The proposed structure which are using for mapping data to OpenMRS its the concept of OpenMRS integration concept for relational mapping database. But, Maimoona this database concept is compitible with sql-based database system.We are using no-sql database so it is not so helpful and work load will not be reduced. To post and process data from submission data to the BaseEntity, Client, Event as well as to get data from these documents both will also extend the codebase.

First time Ziggy saves the data to the HH, ELCO, Mother, Child documents. When HH, ELCO, Mother, Child etc take the services by follow up forms the services data (encouters) are updated to the related Document like HH, ELCO, Mother etc.Suppose for Mother, ANC1, ANC2, ANC3, ANC4, tt1, tt2 etc. can be updated. For EC , fp strategies data can be updated. For Child, Immunizations data can be updated.So , When we need information of any particular registry we can get all history of the Beneficiary in a single query. Actually this is the facility of No-sql based database system. Huge data you can get without relationship mappings.

In the case of the proposed model we have to execute at least three/four/five and so on query with BaseEntity, Client, Event etc depending on the expecting data. If it would be a relational database system in that it would be very helpful, we could get the data using join query.

So, I am not saying to remove your concept but my suggestion is also do not remove ziggy you can apply your concept alongside ziggy. From where can you start I think you can start within route method where handlers are called.

If you think this ziggy library can not fulfill your requirement in that case I can say I will modify ziggy. Because I have experience in NodeJS, AngulaJS based application

[koros]

@mberg @sohelsarder @ndisha @alihabib @raihan-mpower @dimasciput @djazayeri @maimoonak @cagulas I have attached the initial document on this issue, i haven't answered/addressed all issues raised so far but i intend to do that soon https://docs.google.com/document/d/1bBx51sLO5vG7W8NKfDXvGvlqyMiXBXbNkxEMENIjues/edit?usp=sharing

[koros]

@mberg @sohelsarder @ndisha @alihabib @raihan-mpower @dimasciput @djazayeri @maimoonak @cagulas sorry for taking long to respond to the issues raised above, i have added a few more comments on the shared document, i'll continue refining the document as i also complete the proof of concept opensrp-cloudant sync to demonstrate how we can incorporate this technology into opensrp