search

msupply-foundation / open-msupply

Open mSupply represents our most recent advancement in the Logistics Management Information System (LMIS), expanding on more than two decades of development inherited from the well-established legacy of the original mSupply.
https://msupply.foundation/open-msupply/
Other
20 stars 12 forks source link

Initialising a new Open mSupply site with a lot of central server data takes a long time #4658

Open DhanyaHerath opened 3 weeks ago

DhanyaHerath commented 3 weeks ago

What went wrong? 😲

From Mark G:

Why does an Open mSupply store which:

  • Handles items from one master list containing a dozen items
  • Receives stock from 1 supplier
  • Issues to 6 non-store customers Needs to be initialised with several hundred thousand records? This takes literally hours at the end of a the average cellular phone line, and is impractically long in the field. The current work-around is to configuring devices where the internet is better than average, and then take them to the field. We won't get away with this forever. The time will come when we need to initiate a site in the field.

Why must every database have all system data (every item, master list, non-patient name, item-store-join, etc.)? Why can't these details be synced as visibility to them is turned on?

Expected behaviour 🤔

How to Reproduce 🔨

Steps to reproduce the behaviour:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Your environment 🌱

andreievg commented 3 weeks ago

Triage, setting priority as must have to priorities refinement

andreievg commented 3 weeks ago

Needs investigation, what is the breakdown of sync outs for small new sites. The goal is to try and reduce number of records needed to be sent so let's see if we can identify any low hanging fruit. Also discussed trying out connection timeout in reqwest crate and maybe do retries of connection.

andreievg commented 2 weeks ago

During dev test day, me and @lache-melvin had issues initialising tablets, it was very slow and sometimes connection would be stuck for a long time, only to show up with retry. I did some investigation and even put caddy between mSupply and the outside world to setup timeouts, this didn't really have any effect. I added logging and saw that when connection is being 'timed' out, there wasn't even any caddy logs present (so was outside control of central servers).

There is a way to set connection timeout for reqwest, using client builder, while I was trying to set timeouts and replies I stumbled accros gzip header, in reqwest docs looks like it's not automatically turned on, and need gzip features for it to be enabled: https://docs.rs/reqwest/latest/reqwest/struct.ClientBuilder.html#method.gzip

In this branch I experimented with gzip, connection timeout and retries, and batch size increase. I did a few benchmarks connecting to the same central servers on debug build apk and it looks like it's 100-150% faster. I don't think the timeout and connection is helping, because today I never got the 'retry' msg.

Conclusions

Chris-Petty commented 2 weeks ago

I'll start a KDD for the original question - how can we reduce the amount of unnecessary records being sent?