Encounter sync is extremely slow

[vinayvenu]

See attached sync log from the server. While other entity syncs take very little time, save of encounters progressively degrades pretty fast.

App version: 3.38.5

Analysis notes and solution

This is caused by higher number of encounters per subject. In this case, when number of encounters per subject goes to the 100s, save takes a lot of time.

Root cause is the slow implementation of BaseEntity.mergeOn() method. This method takes m parents, looks at all their children and returns one parent with unique values of all their children in it. For m subjects with n encounters each, the merge has a loop that iterates m n n times. m is usually around 1-10 and n can go from 1 to 1000.

The purpose of the merge is to identify unique values in a list of m n items. If we use hashmaps instead of doing a triple loop, this operation can be done with a complexity of O(mn).

https://app.zenhub.com/files/64590403/5bb599a4-6a6c-4a55-80fc-257b65a9e7d3/download

Testing notes

• Sync of
  • Individuals
  • Encounters
  • Program Encounters
  • Checklists
  • Group Subjects
  • Individual Relationships
  • Approvals
  • Tasks
  • Forms (forms, form element groups, form elements, concepts, concept answers)
• Verify that sync brings in data correctly. Push testing is not relevant
• Test for higher number of items to be synced and different organisations
• Fast sync, upload etc testing not relevant

[himeshr]

On evaluation of the proposed solution, we found that the root cause was not due to "high number of JS objects in memory". It was rather due to a convoluted logic in openchs-models BaseEntity.mergeOn() method.

We have simplified merge of children across multiple instances of same parent entity by using a simple HashMap init and overwrite operation. The values of this map are then inserted into the parent entity.

Code changes only in openchs-models.

[vinayvenu]

Moved from avniproject/avni-client#1011 to #45

[vinayvenu]

Modified the card description to match correct analysis and solution

[AchalaBelokar]

• [ ] 1st time sync is taking more than half and hr...which is having more than 1000 encounter it consume near about 45 min
• [ ] No diff org no prob with that sync taking a same time diff between 2 apk is 1 sec or 2 sec