layers and non-geometric tables are sometimes not exported to offline project (network share)

soester commented 5 years ago

preparation

add 7 point-layers from postgres (2 views, 4 tables)
add 1 polygon-layer form postgres (table)
add 29 non geometric tables form postgres (mostly valuelists for relation reference)
configure project for qfield syncronisation (all layers/tables set to offline editing)
export project to offline-folder

issue

Open the exproted project
-> Not all Layers and tables are visible in the layertree (10 non geom Tables, 3 point layers form tables, 2 point layers from views, 1 polygon layer ->> 19 non geom Tables are lost, 1 point layer is lost.)
-> when opening offline data.gpkg all layers are available

we don't know if it is related to our setup or if this is a general bug. we noticed some strange things.

test made so far

wich layers are lost is somehow also depending on layer groups and layer order in layer tree -> its really random wich layers are lost!
Check if it makes a difference if layer is added via brwoser or db manager -> not really, similar issues
started with a clean qgs-file created with 3.4.3 (we thaugth issue was related to file created in 2.18 or 3.2), no widget-config nor layergroups -> same
generate project with modelbaker -> some layers are still not exported

specs of our environment

QGIS version	3.4.3-Madeira	QGIS code branch	Release 3.4
Compiled against Qt	5.11.2	Running against Qt	5.11.2
Compiled against GDAL/OGR	2.3.2	Running against GDAL/OGR	2.3.2
Compiled against GEOS	3.7.0-CAPI-1.11.0	Running against GEOS	3.7.0-CAPI-1.11.0 673b9939
PostgreSQL Client Version	9.6.10	SpatiaLite Version	4.4.0-RC1
QWT Version	6.1.3	QScintilla2 Version	2.10.8
PROJ.4 Version	493

Postgres 9.6 MacOS 10.11.6 qfieldsync 3.0.3

ideas

time out issue dependig on how many tables are in project?
related to not respecting all settings in project-files (as in #89)?
related to #95, #90 ??

contacted @m-kuhn to get some coaching and review of our setup..

soester commented 5 years ago

test with @m-kuhn shows that this is related to geopackage. if qfieldsync is forced to use spatialite this does not occur.

option to chose if geopackage or spatialite is used would be helpful.

amr66 commented 5 years ago

similar issue here. In QGIS i have a geopkg, its features I want to edit in QField. Export creates a data.gpkg, but it does not contain any features. Using a spatialite DB in QGIS, export is still a gpkg, now has features, i can edit and sync back.

soester commented 5 years ago

Retested with QGIS 3.4.6 and 3.6.1 -> Same problems But in Qfieldsync Log following warnings appear:

2019-04-05T08:10:55     WARNING    ERROR: layer not editable
2019-04-05T08:10:56     WARNING    ERROR: layer not editable
2019-04-05T08:10:57     WARNING    ERROR: layer not editable
2019-04-05T08:10:58     WARNING    ERROR: layer not editable
2019-04-05T08:10:59     WARNING    ERROR: layer not editable
2019-04-05T08:11:00     WARNING    ERROR: layer not editable
2019-04-05T08:11:01     WARNING    ERROR: layer not editable

OGR Log:

2019-04-05T08:10:55     WARNING    Cannot find layer od_gaerrestlager_6f4e5c97_98cc_43d9_9f6b_cbc53474ad61.()
2019-04-05T08:10:56     WARNING    Cannot find layer od_kampagne_fdbe8e0a_2dc6_4cd5_a647_ead24d041905.()
2019-04-05T08:10:56     WARNING    Cannot find layer od_messung_messstelle_f26b9c7b_859f_4145_8936_c7df8dad5e8d.()
2019-04-05T08:10:56     WARNING    Cannot find layer od_standort_02360b2f_cd44_47fa_804e_dcd3022ee8cf.()
2019-04-05T08:10:57     WARNING    Cannot find layer od_verwertung_gas_9451228d_259e_43cf_94e6_4fb1fc673861.()
2019-04-05T08:10:57     WARNING    Cannot find layer rel_standortkontakte_651a2c49_49af_4b69_a719_3f7c2a8c5280.()
2019-04-05T08:10:57     WARNING    Cannot find layer vl_atex_zone_9bbbebe3_60cc_4818_bf9b_85256cc09740.()
2019-04-05T08:10:57     WARNING    Cannot find layer vl_auftrag_typ_60767b96_bb35_494f_80ef_78f22096c31b.()
2019-04-05T08:10:58     WARNING    Cannot find layer vl_bew_kat_ch4_b321198b_622b_4e7a_84cf_de3e758b4b8b.()
2019-04-05T08:10:58     WARNING    Cannot find layer vl_bew_kat_schadensausdehnung_7329f48d_e849_4aec_809f_cc6439efb208.()
2019-04-05T08:10:59     WARNING    Cannot find layer vl_bew_kat_zuendquellenabstand_bce4b42b_877a_4293_8a4e_af3a128cef4a.()
2019-04-05T08:10:59     WARNING    Cannot find layer vl_kampagne_status_4227feeb_9ea2_4f9d_aefa_a0afa5b2c0eb.()
2019-04-05T08:10:59     WARNING    Cannot find layer vl_kampagne_typ_b5d449fd_0f3e_4c50_86db_66cacccac6a4.()
2019-04-05T08:11:00     WARNING    Cannot find layer vl_messstelle_art_ee2b617e_d05f_4571_bfef_c9f80c7fb654.()
2019-04-05T08:11:00     WARNING    Cannot find layer vl_messstelle_status_lage_1ef66211_c777_4d07_b018_d3d9f64eb29b.()
2019-04-05T08:11:01     WARNING    Cannot find layer vl_messstelle_teilobjekt_18f5510a_8218_48ec_a1b5_bb9bad10ddc8.()
2019-04-05T08:11:01     WARNING    Cannot find layer vl_messung_status_6300f5f4_514c_41c3_b27f_02be56d1455b.()
2019-04-05T08:11:02     WARNING    Cannot find layer vl_standort_typen_95a02b94_aece_4fa9_ba01_621ca6edd302.()

Problem with locked geopackage?

soester commented 5 years ago

Workaround to let qfieldsync prefer spatialite over gpkg when creating offline project: Change lines 137-156 in 'offline_converter.py' (/Users/sam/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/qfieldsync/core/offline_converter.py on Mac) from:

            try:
                # Run the offline plugin for gpkg
                gpkg_filename = "data.gpkg"
                if self.__offline_layers:
                    offline_layer_ids = [l.id() for l in self.__offline_layers]
                    if not self.offline_editing.convertToOfflineProject(self.export_folder, gpkg_filename,
                                                                        offline_layer_ids,
                                                                        self.project_configuration.offline_copy_only_aoi, self.offline_editing.GPKG):
                        raise Exception(self.tr("Error trying to convert layers to offline layers"))
            except AttributeError:
                # Run the offline plugin for spatialite
                spatialite_filename = "data.sqlite"
                if self.__offline_layers:
                    offline_layer_ids = [l.id() for l in self.__offline_layers]
                    if not self.offline_editing.convertToOfflineProject(self.export_folder, spatialite_filename,
                                                                        offline_layer_ids,
                                                                        self.project_configuration.offline_copy_only_aoi):
raise Exception(self.tr("Error trying to convert layers to offline layers"))

to

            try:
                # Run the offline plugin for spatialite
                spatialite_filename = "data.sqlite"
                if self.__offline_layers:
                    offline_layer_ids = [l.id() for l in self.__offline_layers]
                    if not self.offline_editing.convertToOfflineProject(self.export_folder, spatialite_filename,
                                                                        offline_layer_ids,
                                                                        self.project_configuration.offline_copy_only_aoi):
                        raise Exception(self.tr("Error trying to convert layers to offline layers"))
            except AttributeError:
                # Run the offline plugin for gpkg
                gpkg_filename = "data.gpkg"
                if self.__offline_layers:
                    offline_layer_ids = [l.id() for l in self.__offline_layers]
                    if not self.offline_editing.convertToOfflineProject(self.export_folder, gpkg_filename,
                                                                        offline_layer_ids,
                                                                        self.project_configuration.offline_copy_only_aoi, self.offline_editing.GPKG):
                        raise Exception(self.tr("Error trying to convert layers to offline layers"))

this file can be edited after installation.

@m-kuhn is this a general problem or only related to our setup? if it is general i would try to create a pull request...

soester commented 5 years ago

After doing some testing on ubuntu virtual machine, finaly found the problem: This is only an issue when syncing (export) to shared folder over network. if i set the export path to a local directory everything wors fine.

Tested on Mac and Ubuntu (Virtual) for QGIS 3.6.3 and Qfieldsync 3.1.3

Amphibitus commented 3 years ago

I have the same problem after using a new server. It works fine in the local directory, but some layers are created in the geopackage on the new server but there are no entries in the table. Is there a problem with the rights of the folder?

I disabled the oplocks option on the server and it works fine. It looks like the offline_editing plugin opens the data.gpkg file several times during the process.

Is there any solution to keep using the oplock option continuously?

"OpLocks are made to enable simultaneous file access by multiple users while also improving performance for synchronized caches. In a synchronized cache, when a client requests a file from a server, the shared file may be cached to avoid subsequent trips over the network to retrieve it."

opengisch / qfieldsync