MerginMaps / work-packages

Mergin Maps Work Packages: manage field surveys for multiple teams
MIT License
7 stars 4 forks source link

Inefficient merging of data #61

Closed wonder-sk closed 7 months ago

wonder-sk commented 7 months ago

In make_work_packages(), this bit can be very inefficient for larger files:

geodiff.create_changeset(wp_gpkg_base, master_gpkg_output, wp_changeset_base_output)

it creates big diff files, because the master gpkg has typically much more data than a wp gpkg. This should be avoided, and we should probably just concatenate diffs that we will apply to the master gpkg.

Another source of slowness are list_changes() calls, especially:

        geodiff.list_changes(wp_changeset_base_output, wp_changeset_base_output_json)

... which convert large diff from the previous step to even larger JSONs and that takes a lot of time.