nyalldawson
commented
12 months ago @rouault interesting discussion!
While working on https://github.com/qgis/QGIS-Enhancement-Proposals/issues/257 (a SensorThings provider) , I've been wondering if we need some generic / cross provider approach to handle feature relationships in a more flexible way. Currently relationships in QGIS (as you've noted) are tied heavily to the concept of individual layers, but this doesn't translate well to data models like you've described here or the SensorThings model where the relationship structure is more complex and tightly tied into the data model itself.
Another example would be handling "attachments" from an ArcGIS REST server. These aren't available as a standard layer, and indeed can't even be retrieved in bulk. Rather you need to call a specific "get attachments" API per feature that you want to retrieve the attachments for. Again, this setup doesn't fit in well with the current layer based approach to relationships.
So I'm wondering if we shouldn't use this opportunity to instead develop a "get related features" concept within the vector data provider itself, which would allow for interactive feature-based (and nested) related feature retrieval.
I'm keen to hear your thoughts... it's quite a different approach to the "flattening" technique you've proposed here, but might ultimately be a better fit for this data model...
rouault
wonder-sk
kraftto
QGIS Enhancement: Support for Complex Features in WFS provider
Date 2023/11/13
Author Even Rouault (@rouault)
Contact even.rouault at spatialys.com
Maintainer @rouault
Version QGIS 3.36 or 3.38
Sponsored by QGIS-DE user group (QGIS Anwendergruppe Deutschland e.V.)
Summary
The QGIS WFS provider (WFS client) supports WFS 1.0, 1.1 and 2.0 protocols, but it is currently restricted to consuming features returned as GML simple features, that is features that have attributes of simple pre-defined types (string, integer, floating-point number, datetime). While WFS servers serving GML simple features are common, there are various data models around that don't fit into that model and use more complex GML schemas, where feature properties can be repeated or be made of nested XML constructs. This is typically the case for Inspire. Currently, QGIS rejects such WFS layers. While there are workarounds, using the WFS 2.0 client plugin or the QGIS GML Application Schema Toolbox plugin, they require the user to be aware of their existence, and don't provide the same level of user experience than when working with a simple feature WFS layer. Hence this proposal of enhancing the existing WFS provider to be able to deal with complex feature schemas. The implementation will expose properties of complex types as JSON content, converted from XML.
Proposed Solution
Let's state it in a straighforward way: dealing with WFS/GML complex features is a pain and based on past experience developing the OGR GMLAS - Geography Markup Language (GML) driven by application schemas - driver I don't believe there is an ideal solution. The solution proposed here is both a compromise from an implementation and usability point of view. The choice of exposing complex properties as a JSON serialized string from nested XML is what makes it possible to use the existing infrastructure of the WFS provider. The other alternative that could be imagined, and which was taken by the OGR GMLAS driver, would have been to expose a (complex) relational data model of many simple feature tables with lots of relationships between them. It woulnd't fit at all as a QGIS provider, as a QGIS provider is bound to a single layer, and such output is not necessarily easy to manipulate by end users. Hence the QGIS GML Application Schema Toolbox plugin is not totally deprecated by the enhanced WFS provider, in use cases where a relational view of the data model is needed.
Concretely, the QgsWFSProvider::readAttributesFromSchema() method which analyzes the XML schema returned by the WFS DescribeFeatureType request on the layer of interest will be modified to fallback to the OGR GMLAS driver when the schema is not natively understood by QGIS (that is, it is not simple features). More precisely, only the schema analysis capability of the GMLAS driver will be used, by querying the special
_ogr_fields_metadatalayer which returns the properties of a FeatureType and their nature (if they are simple or complex, their cardinality). To be noted too that the GMLAS driver automatically "flattens" the data structure, even when the content is nested, but when the constraints schema show that there is a 1:1 cardinality (cf below example where ps:inspireID/base:Identifier/base:localId can be flattened asinspireid_identifier_localid). The QgsGmlStreamingParser class will be modified to use the hints (mapping between XPath and QGIS fields) provided by the GMLAS schema analyzer to identify properties. A QgsXML class will be added to offer the translation from XML to equivalent JSON.For server-side filtering of simple content that is not at the first level of nesting, QgsOgcUtils::expressionToOgcExpression() and QgsOgcUtils::SQLStatementToOgcFilter() will need to be enhanced to map QGIS field names to proper XPath.
The first time the OGR GMLAS driver needs to access a remote schema, it downloads its content and caches the file locally. However with complex schemas, this can involve downloading several tens of files in a cascaded way, which can take several tens of seconds depending on network and server speed. The OGR GMLAS driver will be modified to allow cancellation of the downloading of files, and the QGIS WFS provider, when invoked from QGIS graphical user interface, will cause the OGR GMLAS driver to be invoked in a background task that can be canceled to avoid blocking the user interface. The OGR GMLAS driver will also be modified to use the the pluggable CPL networking capability of CPLHTTPPushFetchCallback(), to be able to use QGIS networking infrastructure to download the XML schemas.
The WFS provider automated tests will be enhanced to test for the above mentioned (non UI-based) changes/enhancements.
Example
Let's look at the output of a Inspire data model: https://inspire.brandenburg.de/services/ps_schutzg_wfs?service=WFS&version=2.0.0&typename=ps:ProtectedSite&request=getfeature&count=1
Querying the structure of the protectedsite layer with the OGR GMLAS driver yields:
The layer fields exposed by QGIS will be:
Affected Files
In QGIS:
In GDAL:
Limitations
Performance Implications
No measurable performance hit expected on simple features WFS layers.
For complex features WFS layers, an initial delay of several seconds the first time a new schema is read is expected.
Backwards Compatibility
There should be no backward compatibility issues as this is a new feature.
Issue Tracking ID(s)
#27076 - WFS 2.0 complex features not supported #52227 - Error loading WFS
Votes
To be done