Improve QGIS project loading time - mainly for QGIS Server

[rldhont]

QGIS Enhancement: Improve QGIS project loading time - mainly for QGIS Server

Date 2020/06/03

Author DHONT René-Luc (@rldhont)

Contact rldhont at 3liz dot com

maintainer @rldhont

Version QGIS 3.16

Summary

The trust project option was created to try to improve the loading time of a QGIS project for the postgres layers.

This option forced 2 things for the postgres layers: not checking the uniqueness of the primary key and using the extent of the layer defined in the project.

Since QGIS 3, the trust option forces the use of the layer extent defined in the project for all vector layers and modifies the checkPrimaryKeyUnicity parameter of the Datasource of the postgres layers.

There is also for some layers a Datasource parameter useEstimatedMetadata allowing to accelerate the retrieval of information on the layer (grip, feature count, layer type, etc).

To improve the loading of a QGIS project, you should:

1. Extend the use of the extent defined in the project to the Raster layers mReadExtentFromXml
2. In the properties of the layers, have a dedicated configuration interface: definition of the extent, useEstimatedMetadata, checkPrimaryKeyUnicity, etc.
3. Create new parameters: checkGeometryType, checkSrid, etc.

Some providers will be more impacted than others like postgres and oracle.

Proposed Solution

Extend the use of the extent defined in the project

For vector layers, we have:

QgsVectorLayer::QgsVectorLayer( const QString &vectorLayerPath,
                                const QString &baseName,
                                const QString &providerKey,
                                const QgsVectorLayer::LayerOptions &options )
  : QgsMapLayer( QgsMapLayerType::VectorLayer, baseName, vectorLayerPath )
  , mServerProperties( new QgsVectorLayerServerProperties( this ) )
  , mAuxiliaryLayer( nullptr )
  , mAuxiliaryLayerKey( QString() )
  , mReadExtentFromXml( options.readExtentFromXml )

We would have to add readExtentFromXml to QgsRasterLayer::LayerOptions struct, a private attribute mReadExtentFromXml and make the code to take it into account inQgsRasterLayer.

Management interface

In the Layer Properties window, in the Source tab, we must add a management interface for the extent defined in the project, but also for Datasource parameters such as useEstimatedMetadata and checkPrimaryKeyUnicity.

Management of the extent defined in the project

If it is possible to use the extent defined in the project for all the layers, it is necessary to be able to manage this extent.

For the moment QGIS only offers a button to calculate the extent of the layer from the data.

This extent can evolve by adding data, it would therefore be necessary for certain layers to regularly recalculate this extent or define a maximum extent integrating the maximum of data.

We propose to define an interface displaying the current extent as for the extent of the CRSes and the values of this extent.

We also propose to define several buttons allowing to carry out the following operations and reuse QgsExtentWidget:

• Calculate the data extent (current button)
• Use the extent of a layer
• Use the current canvas extent
• Manually modify the values of the extent

Managing Datasource settings

Datasource parameters can no longer be modified once the layer has been created:

• useEstimatedMetadata to activate the use of the estimated metadata table
• the primary key and the verification of its uniqueness checkPrimaryKeyUnicity.
• selection by feature ID selectAtIdDisabled()
• the geometry and bounding box column
• the new parameters

Define new parameters to speed up the loading of layers

By studying the source codes of providers postgres and oracle, it is possible to define new Datasource parameters that would improve the loading of the project:

• checkGeometryType, checking the geometry type from the data source
• checkSrid, checking the coordinate system from the data source

Example(s)

(optional)

Affected Files

Extend the use of the extent defined in the project

• src/core/raster/qgsrasterlayer.h
• src/core/raster/qgsrasterlayer.cpp
• src/core/qgsproject.cpp

Management interface

• src/gui/vector/qgsvectorlayerproperties.cpp
• src/gui/vector/qgsrasterlayerproperties.cpp

Define new parameters to speed up the loading of layers

• src/gui/vector/qgsvectorlayerproperties.cpp
• src/providers/oracle/qgsoracleprovider.cpp
• src/providers/postgres/qgsoraclefeatureitertor.cpp
• src/providers/postgres/qgspostgresprovider.cpp
• src/providers/postgres/qgspostgresfeatureitertor.cpp

Performance Implications

Hope to speed up the loading of layers

Further Considerations/Improvements

Better managing layer's datasource

Backwards Compatibility

none

Votes

(required)

[rldhont]

Open for feedback @andreasneumann @pblottiere @elpaso @mdouchin @dmarteau

[rldhont]

First related PR:

• New project reading flag to trust layer metadata with server settings
• New QGIS Server setting environement to disable GetPrint and do not load layouts

[mdouchin]

Thanks for the work. What about proposing a solution to disable also:

• the checking of editing capabilities: server side, it is only useful when WFS-T related checkboxes are checked.
• The checking of every feature field type and other properties: let's really trust the project here too ?

[borysiasty]

The checking of every feature field type and other properties: let's really trust the project here too ?

Btw. don't miss the recent speedup here: https://github.com/qgis/QGIS/pull/38158 . With only checking the relevant fields, it works smoothly in large databases in the end.

[rldhont]

The checking of every feature field type and other properties: let's really trust the project here too ?

Btw. don't miss the recent speedup here: qgis/QGIS#38158 . With only checking the relevant fields, it works smoothly in large databases in the end.

Thanks @borysiasty. this is for postgres layers. Does the same has been done for oracle or other database ?

[rldhont]

The trust flag only skips the extent request to the provider. For you what are the other requests to skip with this flag ?

[rldhont]

To enhance the trust flags: Propagate trust layer metadata flag https://github.com/qgis/QGIS/pull/38464

[borysiasty]

Thanks @borysiasty. this is for postgres layers. Does the same has been done for oracle or other database ?

Nope. I only touched the Postgres provider. I guess that particular issue with fetching superfluous metadata was postgres-specific.