search

inspire-eu-validation / ets-repository

Repository for Executable Test Suites of the INSPIRE validator (under development)
European Union Public License 1.2
13 stars 25 forks source link

Unexpected test failure: no AdministrativeUnit features found #142

Open PeterParslow opened 6 years ago

PeterParslow commented 6 years ago

Hi, I ran a test of a sample output from our new 'Administrative Units' production system, and was surprised to fail on au-gml.a.1 - it didn't find the one feature in the sample.

This may be related to failing gml.a.2, because we deliberately use a different root element. The reason we use our own Feature Collection is documented at https://www.ordnancesurvey.co.uk/docs/policies/gml-design-policy.pdf (section 3.2). I advised our supplier not to use the deprecated SpatialDataSet or gml:FeatureCollection.

But I had hoped that it would go on & validate the features within our collection.

Results at http://inspire-sandbox.jrc.ec.europa.eu/etf-webapp/v2/TestRuns/EID0e187bcd-1b27-42a0-8027-6b186626614e.html

Assertion URI: http://inspire-sandbox.jrc.ec.europa.eu/etf-webapp//v2/TestRuns/EID0e187bcd-1b27-42a0-8027-6b186626614e.html?lang=en#EID5c3747c0-ff39-4891-9213-45af8bbc746d

Uploaded sample zipped & attached: BdyLine_Product_INSPIRE_Example_v4.zip

Peter

cportele commented 6 years ago

Hi Peter - yes, the reason is the root element.

As stated in https://github.com/inspire-eu-validation/data-encoding/blob/3.3/inspire-gml/basic.md and the test report, only wfs:FeatureCollection (WFS 2.0), gml:FeatureCollection (GML 3.2) or base:SpatialDataSet (Base schema 3.2 or 3.3) are recognized. The use of wfs:FeatureCollection is recommended.

This approach is necessary as it is unclear how to select all features in an XML document with an unknown root element. Yes, //schema-element(gml:AbstractFeature) would do the trick in an XPath 2.0 implementation, but for performance reasons XML databases are typically not schema-aware and do not support this XPath 2.0 function in XQueries. BaseX is no exception.

If there is agreement in the MIG-T that this should be relaxed (I will tag this issue for discussion in the MIG-T subgroup), then another approach to identify all features in the XML documents is needed.

One aspect to consider in this is also that if the validator cannot identify the features in an XML file, other software may also have problems. This is why D2.7 recommends the use of wfs:FeatureCollection.

PeterParslow commented 6 years ago

Clemens, I suspected as much.

The problem, as I see it, is that wfs:FeatureCollection doesn’t obey the rules of GML Feature Collections. We have other specific issues with it too; our data customers (& their software suppliers) find it useful if the feature collection has a boundingBox, and we find it useful to provide metadata (e.g. description, publication date). “proper” GML feature collections allow these; the wfs one doesn’t.

The other two options have been deprecated within their respective schemas.

I recognise the D2.7 Recommendation, but hadn’t spotted it being ‘upgraded’ to a requirement. I’d be happy to present more detail for MIG-T, if that would be useful.

Peter

From: Clemens Portele [mailto:notifications@github.com] Sent: 06 February 2018 16:31 To: inspire-eu-validation/ets-repository ets-repository@noreply.github.com Cc: Peter Parslow Peter.Parslow@os.uk; Author author@noreply.github.com Subject: Re: [inspire-eu-validation/ets-repository] Unexpected test failure: no AdministrativeUnit features found (#142)

Hi Peter - yes, the reason is the root element.

As stated in https://github.com/inspire-eu-validation/data-encoding/blob/3.3/inspire-gml/basic.md and the test report, only wfs:FeatureCollection (WFS 2.0), gml:FeatureCollection (GML 3.2) or base:SpatialDataSet (Base schema 3.2 or 3.3) are recognized. The use of wfs:FeatureCollection is recommended.

This approach is necessary as it is unclear how to select all features in an XML document with an unknown root element. Yes, //schema-element(gml:AbstractFeature) would do the trick in an XPath 2.0 implementation, but for performance reasons XML databases are typically not schema-aware and do not support this XPath 2.0 function in XQueries. BaseX is no exception.

If there is agreement in the MIG-T that this should be relaxed (I will tag this issue for discussion in the MIG-T subgroup), then another approach to identify all features in the XML documents is needed.

One aspect to consider in this is also that if the validator cannot identify the features in an XML file, other software may also have problems. This is why D2.7 recommends the use of wfs:FeatureCollection.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/inspire-eu-validation/ets-repository/issues/142#issuecomment-363480160, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIGUoX_72ITBz2MMAEHbqKHrq0p9ksu0ks5tSH5EgaJpZM4R7F2C.

This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.

Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.

Thank you for your cooperation.

Ordnance Survey Limited (Company Registration number 09121572) Registered Office: Explorer House Adanac Drive Southampton SO16 0AS Tel: 03456 050505 http://www.os.uk

cportele commented 6 years ago

OK, I understand. Whatever the approach is that the MIG-T prefers, a reliable mechanism to identify the features is needed (for the validator and everyone else).

Just to avoid confusion: It is a recommendation in D2.7, not a requirement. Which is why the other two collection types are currently recognised, too, as we had seen them used with INSPIRE data.

PeterParslow commented 6 years ago

It was cheeky of me to say "requirement"; it's only become a "requirement" if trying to use the validator.

For information, our corporate GML design choices were heavily influenced by OGC Engineering Report 14-057 "UK Interoperability Assessment Plugfest" (which we sponsored). This included a number of exercises that included our 'custom' feature collections. I need to do a bit more checking that we've actually fixed the various issues uncovered, but it did demonstrate that all the software that was used in the plugfest could cope with 'user defined' feature collections (and showed the preference for boundingBox).

Peter

From: Clemens Portele [notifications@github.com] Sent: 06 February 2018 16:53 To: inspire-eu-validation/ets-repository Cc: Peter Parslow; Author Subject: Re: [inspire-eu-validation/ets-repository] Unexpected test failure: no AdministrativeUnit features found (#142)

OK, I understand. Whatever the approach is that the MIG-T prefers, a reliable mechanism to identify the features is needed (for the validator and everyone else).

Just to avoid confusion: It is a recommendation in D2.7, not a requirement. Which is why the other two collection types are currently recognised, too, as we had seen them used with INSPIRE data.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/inspire-eu-validation/ets-repository/issues/142#issuecomment-363488451, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIGUoURfY0hGwLzwMGzrSUhbAC9bX0Zhks5tSIOHgaJpZM4R7F2C.

This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.

Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.

Thank you for your cooperation.

Ordnance Survey Limited (Company Registration number 09121572) Registered Office: Explorer House Adanac Drive Southampton SO16 0AS Tel: 03456 050505 http://www.os.uk

cportele commented 6 years ago

We had to make it a "requirement" make all the other requirements testable. That said, if https://github.com/interactive-instruments/etf-webapp/issues/37 would be implemented, this would provide the capability to determine all features in XML documents and this test would no longer be needed.

But it is good news that the software in the OS plugfest obviously were all schema-aware (or maybe they just ignored the namespace and simply looked at the local element names FeatureCollection / featureMember, which could also be another convention to support in the validator).

heidivanparys commented 6 years ago

GDAL/OGR seems to look at the local element names only:

michellutz commented 6 years ago

[2017.4 meeting 2018-02-15] This seems to be an ETF implementation issue (that could be fixed e.g. by implementing interactive-instruments/etf-webapp#37 as suggested by @cportele ). Since this test does not reflect a requirement, but is simply a pre-requisite for identifying features in the XML document, the test should be as relaxed as possible. But at the same time, some guidance would be useful also for other client applications, so they know what to expect. The solution used in GDAL/OGR seems to be a useful compromise. Therefore it is suggested to mirror this solution in the test (at least temporarily) and to document this in a corrigendum to D2.7 (section B.3.3 Root element of the exchange document) should be proposed.

PeterParslow commented 6 years ago

It will be interesting to draft that corrigendum; I've been looking a bit more into the OGC definition of "feature collection", bearing in mind some of the findings of an OGC plugfest a few years back, in which just one of the software providers identified bugs in a number of OS feature collections (see http://www.opengeospatial.org/projects/initiatives/ukiap)

It turns out that GML 3.2.1 has one definition for a feature collection (i.e. short set of requirements that a feature type must satisfy to be considered a feature collection); GML Simple Features over rides it, giving a different definition (in order to rule out nested hierarchies of feature collections). The WFS Feature Collection does not satisfy either set of requirements.

michellutz commented 6 years ago

[2017.4 meeting 2018-03-16] The implementation of interactive-instruments/etf-webapp#37 could be useful. Independently, the solution used in OGR/GDAL to find features in feature collections should be proposed as a corrigendum to the INSPIRE Encoding Guidelines (D2.7). If accepted by the MIG, the ATS/ETS should be updated accordingly.

michellutz commented 6 years ago

[2017.4 meeting 2018-04-19] The proposed TG corrigendum is tracked on the MIG collaboration platform in https://ies-svn.jrc.ec.europa.eu/issues/3218

michellutz commented 6 years ago

The proposed TG corrigendum did not receive any feedback, so it should be implemented in the TGs, ATS and ETS.