search

opengisch / qfieldcloud

QFieldCloud allows seamless synchronization of your field data with your spatial infrastructure with change tracking, team management and online-offline work capabilities in QField.
https://qfield.cloud
MIT License
102 stars 44 forks source link

Speed up packaging #894

Closed nirvn closed 3 months ago

nirvn commented 4 months ago

This PR reworks the packaging job workflow to prevent 3 project loads and instead rely on a single initial load. This can lead to massive speed gains on complex projects and/or projects containing remote sources that are slow.

Requires the following libqfieldsync improvements: https://github.com/opengisch/libqfieldsync/pull/68

suricactus commented 4 months ago

The decision to load the project 3 times was by design to make each step in a Job workflow independent from the rest. Now they will become tightly coupled and "load project" becomes mandatory. Do we have any specific proof how much do we gain for such an important change that is introduced here?

+1 in putting the logic in a single function, but I would prefer to still call it three times, unless clear evidence we really save 20+ seconds on this.

nirvn commented 4 months ago

@suricactus , the speed gains are easy to replicate. Just take a QGIS project that loads in 10 seconds, then multiply that by 3 to get the time needlessly spent re-loading it 3 times when we can load it only once.

For one of our clients - the Groupements Forestiers Quebec - , a packaging job takes ~5 minutes, each project load taking 1 minute or so on the cloud server. I can provide job IDs, but I think we already looked at one together yesterday.

nirvn commented 4 months ago

Avoiding the CPU cost of reloading a project three times is also not negligible, that surely reduces stress on the server infrastructure. Project loading is costly unless you have a single geopackage + an OSM basemap layers (e.g., every time you load the project, you re-create layouts, relationships, 2D and 3D map views, refetch metadata of layers for some data providers, etc.).

Here's a packaging that lasted >4 minutes: https://app.qfield.cloud/a/ggeneau/octobre_2023/jobs?id=91a62569-2a24-454a-b9eb-53ea66292374&job-preview=1

Different project, same ~4 minutes issue: https://app.qfield.cloud/a/superviseur_gfca/Projet_PASCAL_20231208/jobs?id=9b020b46-d561-42d8-aa6f-ea8919868a06&job-preview=1

suricactus commented 4 months ago

Would be great to monitor back the actual speed improvement of these jobs. I have the gut feeling it will be negligible.

nirvn commented 4 months ago

@suricactus , PR adjusted to match the content of our discussion yesterday.