Crash during mobile db migration for large fast sync dbs

mahalakshme commented 7 months ago

@petmongrels how long does it take for large databases to complete migration?

mahalakshme commented 7 months ago

Vivek replied upto a minute

mahalakshme commented 7 months ago

Some of the Cases to test:

Should not probably take more than one minute
Pressing back/killing the app when upgrading
old fast sync
new upgradation from old version app
with different large data sets
Avni icon display when upgrading
Make sure all data is there as expected after migration - observations, location, regex validations in first name etc.,

AchalaBelokar commented 7 months ago

[ ] I login with AchalaB@Calcutta_kids and then set a fast sync and then upgrade the app. I started upgrading and then it is sowing mw upgrading data please do not close the app. 3.21 to 3.38 still showing this msg.
[ ] After killing the app same message is showing for long time.
[ ] I Login with AchalaB@sncu and updated the app ...then I kill the app and after open it is showing me white screen app is not responding

petmongrels commented 6 months ago

Can you measure how long it taking you. It will finish.

On Thu, Feb 22, 2024, 3:42 PM AchalaBelokar @.***> wrote:

I login with @.***_kids and then set a fast sync and then upgrade the app. I started upgrading and then it is sowing mw upgrading data please do not close the app. 3.21 to 3.38 still showing this msg.

After killing the app same message is showing for long time.

— Reply to this email directly, view it on GitHub https://github.com/avniproject/avni-client/issues/1309#issuecomment-1959116471, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAZ3C73QV76YEKXRHT5U7DYU4KYRAVCNFSM6AAAAABDQSZOVSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZGEYTMNBXGE . You are receiving this because you were mentioned.Message ID: @.***>

petmongrels commented 6 months ago

It will be better perhaps to implement a progress bar on upgrade screen and similar on fast sync restore. Other wise it may cause lot of confusion. Keep awake may be required if not there

On Fri, Feb 23, 2024, 9:00 AM Vivek Singh @.***> wrote:

Can you measure how long it taking you. It will finish.

On Thu, Feb 22, 2024, 3:42 PM AchalaBelokar @.***> wrote:

I login with @.***_kids and then set a fast sync and then upgrade the app. I started upgrading and then it is sowing mw upgrading data please do not close the app. 3.21 to 3.38 still showing this msg.

After killing the app same message is showing for long time.

— Reply to this email directly, view it on GitHub https://github.com/avniproject/avni-client/issues/1309#issuecomment-1959116471, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAZ3C73QV76YEKXRHT5U7DYU4KYRAVCNFSM6AAAAABDQSZOVSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZGEYTMNBXGE . You are receiving this because you were mentioned.Message ID: @.***>

mahalakshme commented 6 months ago

Yeah agreed it will be good to show progress bar else users might think the app is stuck.

petmongrels commented 6 months ago

Re Compaction

113k ProgEnc in all cases.

Server obs count queries

`select count() from individual i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.observations); -- 14255 select count() from program_encounter i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.observations); -- 786550 select count() from program_encounter i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.cancel_observations); -- 18670 select count() from program_enrolment i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.observations); -- 12023 select count() from program_enrolment i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.program_exit_observations); -- 2624 select count() from encounter i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.observations); -- 46860 select count() from encounter i join address_level on i.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.cancel_observations); -- 0 select count() from checklist_item i join checklist on i.checklist_id = checklist.id join program_enrolment on checklist.program_enrolment_id = program_enrolment.id join address_level on program_enrolment.address_id = address_level.id and address_level.title = '1' cross join lateral jsonb_object_keys(i.observations); -- 18828

-- 14255+786550+18670+12023+2624+0+18828 = 852950`

DB file sizes

avni/db$ ls -ltS total 1544328 -rw-r--r--@ 1 vsingh staff 312098816 Feb 26 13:21 default.7.0.without.compact.realm -rw-r--r--@ 1 vsingh staff 187318272 Feb 26 13:21 default.6.1.without.compact.realm -rw-r--r--@ 1 vsingh staff 185421824 Feb 26 13:58 default.6.1.with.compact.realm -rw-r--r--@ 1 vsingh staff 102535168 Feb 26 13:44 default.7.0.with.compact.realm

Outcome

Running migration is increasing the database size due to copying I think (going to little less than double the size)
Compacting after migration reduces size due to two reasons
- 7.0-without-compact is taking more space because the freed up space has not been reclaimed yet so it includes the space taken by the deleted schemas + the new embedded objects
- secondly due to reduced number of observations overall. Number of observations for this catchment from server side = 8,52,950. In 6.1 realm file the number of observations are = 18,55,479.
So the net net gain is due to 10,00,000 less observations.

himeshr commented 6 months ago

Shouldn't we compact only when there is lot of unused space in the file.? https://www.mongodb.com/docs/realm/sdk/react-native/realm-files/compact-realm/#realm-configuration-file

AchalaBelokar commented 6 months ago

[x] After downloading the 6.1 apk try to login with AchalaB@Calcutta_kids it is showing this message.. After performing slow sync app is crashed .. blank screen is coming

[x] And after killing the app and reopen app still app is showing white screen

himeshr commented 6 months ago

Deleted fast-sync file, fresh sync should work now.

AchalaBelokar commented 6 months ago

[ ] I login with AchalaB@sncu and created one individual and after that set up fast sync and update the app. for updating the app it is taking more than 5 min its like 6,7 min .. so this msg is misleading to user.

[ ] 1st time login with the AchalaB@calcutta_kids and shift to another app and after that come to avni app it is showing me same msg may take up to 5 min start time is 4.22 and end it 4.30

AchalaBelokar commented 6 months ago

[x] While updating the aap I kill the app then it will start from time we reopen the app.
[x] After pressing back button it is not working as expected
[x] I created a individual and sync app after that I updated the app all data is visible on app.
[x] Back button is not working while updating the app
[x] I am able to see the avni logo with message while updating the app.
[x] observation which I created it is reflected as we are expected .

avniproject / avni-client