Pre-defined ATOM: OpenSearch Describe Spatial Data Set operation bugs

[mpleic]

Hello,

there are bugs in the Pre-defined ATOM ETS related to validation of the Describe Spatial Data Set operation.

By the TG Requirement 42 and the specified XPath expression of the element in the ATS description the "Url" element representing the Describe Spatial Data Set operation

• must have an attribute "rel" with the value "describedby"
• does not implement the "crs" parameter, especially not as a mandatory parameter.

Therefore, "describeSpatialDataSetUrlTemplates" elements

• in the line L1440 should be queried by the "@rel='describedby'" condition (like in the line L1323)
• in lines L1324 and L1441 should not be queried by the "contains(@template,'crs')" condition.

Thank you in advance. Marko Pleić

[mpleic]

I made those changes and tested ETS locally. After making those changes I have noticed the test step "Query examples for each dataset" (Source: Abstract Test Case 'OpenSearch Description Query examples for each dataset') is not working as expected. Test step soruce code from the line L1509 to L1600 doesn't fit the description:

• "xmlHolder" is a "Describe Spatial Dataset" response which is an ATOM feed XML, therefore it can't query "getSpatialDataSetUrlTemplates" elements on it on the line L1522
• there is no "category CRS" elements parser from an ATOM feed XML returned by the "Describe Spatial Dataset" operation and it doesn't create and test a Get Spatial Dataset url with the CRS for each category CRS?

Can you please review and fix this test step?

[mwjsanders]

We are facing the same problem when validating our ATOM services and are curiously awaiting a fix

[fabiovin]

Dear @mpleic,

sorry for the late reply.

We are analyzing the issue, in the meantime could you please send us the ATOM you used, in order to reproduce and verify the issue.

Thanks, Fabio

[mpleic]

I would have sent it before, but unfortunately the ATOM service we used is not yet publicly available. We developed it, but we did not receive an official order from the client to do so.

[dperezBM]

Dear @mpleic,

In order to replicate what is causing this error, could you please send us the data you are using? If the information contained in this data is sensitive, please send it to inspire-helpdesk@bilbomatica.com, it will be treated with maximum privacy and will be deleted once the error is solved.

Best regards.

[arantzaetxebarria]

Dear @mpleic

We have implemented some changes related to the issue https://github.com/INSPIRE-MIF/helpdesk-validator/issues/819, and we think this issue may have been solved. Could you please verify?

Best regards

[mpleic]

Dear @arantzaetxebarria

We've run validation process initially at the staging instance, and it failed at the "Iterate over Get Spatial Dataset URL" reporting the following error message: "Expected a 'Content-Type' header field to be returned by the server." altough every valid Get Spatial Dataset with the valid Accept header media type returns the requested Content-Type.

For example: curl --header "Accept: application/x-gmz" -s -I "https://geoportal.dgu.hr/services/atom/search?crs=EPSG%3A3035&language=hr&spatial_dataset_identifier_code=1123&spatial_dataset_identifier_namespace=http%3A%2F%2Fgeoportal.nipp.hr%2Fhr%2Fnipp%2Fpp%2F" > download.txt

Returns the following response header: HTTP/1.1 200 OK Server: Apache-Coyote/1.1 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache Expires: 0 X-Frame-Options: DENY X-Application-Context: application:prod Content-Disposition: attachment; filename=INSPIREBuilding(BU-CORE2D).zip Content-Type: application/x-gmz Content-Length: 108316489 Date: Thu, 15 Jun 2023 12:51:27 GMT

We've fetched those changes and tested our service locally. By viewing the logs we found out there are some generic errors while making the requests which seems to be invalid:

2023-06-15 12:21:00.075 [Thread-100] ERROR c.e.s.SoapUI - An error occurred [Invalid query], see error log for details 2023-06-15 12:21:00.076 [Thread-100] ERROR s.errorlog - org.apache.commons.httpclient.URIException: Invalid query org.apache.commons.httpclient.URIException: Invalid query at org.apache.commons.httpclient.URI.parseUriReference(URI.java:2112) at org.apache.commons.httpclient.URI.(URI.java:177) at com.eviware.soapui.impl.wsdl.submit.filters.EndpointRequestFilter.filterAbstractHttpRequest(EndpointRequestFilter.java:45) at com.eviware.soapui.impl.wsdl.submit.filters.AbstractRequestFilter.filterRequest(AbstractRequestFilter.java:33) at com.eviware.soapui.impl.wsdl.submit.transports.http.HttpClientRequestTransport.sendRequest(HttpClientRequestTransport.java:172) at com.eviware.soapui.impl.wsdl.WsdlSubmit.run(WsdlSubmit.java:119) at com.eviware.soapui.impl.wsdl.WsdlSubmit.submitRequest(WsdlSubmit.java:79) at com.eviware.soapui.impl.support.http.HttpRequest.submit(HttpRequest.java:243) at com.eviware.soapui.impl.wsdl.teststeps.HttpTestRequestStep.run(HttpTestRequestStep.java:263) at com.eviware.soapui.impl.wsdl.support.AbstractTestCaseRunner.runTestStep(AbstractTestCaseRunner.java:211) at com.eviware.soapui.impl.wsdl.testcase.WsdlTestCaseRunner.runCurrentTestStep(WsdlTestCaseRunner.java:47) at com.eviware.soapui.impl.wsdl.support.AbstractTestCaseRunner.internalRun(AbstractTestCaseRunner.java:138) at com.eviware.soapui.impl.wsdl.support.AbstractTestCaseRunner.internalRun(AbstractTestCaseRunner.java:46) at com.eviware.soapui.impl.wsdl.support.AbstractTestRunner.run(AbstractTestRunner.java:129) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

The ATOM service we developed is publicly available since a week ago, so you can use it for the testing purposes now without any access limitations or restrictions on use: https://geoportal.dgu.hr/services/atom/xml

Regards, Marko Pleić

[arantzaetxebarria]

Dear @mpleic

Thanks for your tests and feedback, we think the validator in staging environment maybe is giving this error because the URL https://geoportal.dgu.hr/services/atom/search?crs=EPSG%3A3035&language=hr&spatial_dataset_identifier_code=1123&spatial_dataset_identifier_namespace=http%3A%2F%2Fgeoportal.nipp.hr%2Fhr%2Fnipp%2Fpp%2F returns a 404 error code

Regards

[mpleic]

Dear @arantzaetxebarria

The URL returns 404 error code because the validator is sending generic "Accept" header in a request: text/html, image/gif, image/jpeg, ; q=.2, /*; q=.2 The backend of service expects the "Accept" header always to be sent in a request providing the exact format of a predefined dataset file, because there can be different file formats served, for example "Accept: application/x-gmz" as I mentioned before.

Then

• we made an exception in that case, making the backend returning the predefined dataset file with the default file format
• we provided Get Spatial Data Set templates for all of the available data sets in the OpenSearch description document as requested in Comparison of number of query elements and entry elements test step (ID EID129238a2-999e-4f02-a4e2-63a0d718cc9f) giving the following error message: There are '5' Entry elements in the Dataset feed that are not referenced in Query elements in Opensearch. Found '[0449, 1125, 1128, 1124, 1127]' Entry elements codes without Query code reference.

Even in that case validation process failed checking other data sets.

For example, "Sustavi geografskih mreža" dataset is avaliable only in the ESRI shapefile file format (application/vnd.shp) and EPSG:3765 CRS, but the validator expects "application/x-gmz" format since it is specified through the URL template.

According to the Technical guidelines for the ATOM service implementation, there can be single (TG Requirement 28, 29) or even multiple files (TG Requirement 31) per predefined ATOM dataset, and generally each of them can be in different file formats, and the spatial data can be in the different coordinate referece systems (TG Requirement 26).

Except for the above, according to the TG: When a dataset is not downloadable as a single file, but is a multipart dataset, then the OpenSearch result for the Get Spatial Dataset operation allows for multiple results to be returned. In order to avoid an HTTP multipart response, which is not supported by browsers and would require specific clients, the Get Spatial Dataset operation shall return an Atom feed containing the links to the files to be downloaded instead of the files themselves as in the Example 10: OpenSearch URL template returning an Atom feed.

This implies the content-type to be application/atom+xml in a response header when a multiple predefined dataset is requested.

Therefore, there can be multiple Get Spatial Data Set templates for every available predefined dataset format type in ATOM service, including application/atom+xml file format additionally if there is at least one multiple predefined dataset present in ATOM service. But that doesn't mean that all predefined datasets are available in all of those file formats and CRSs combinations.

Therefore, in the context of the validator, the correct approach would be to find at least one valid response i.e. valid content-type in the response header specified in the accept header of a request. In other words, it is necessary to iterate over all available file types specified in the "template" URL elements, with the inspire_dls:spatial_dataset_identifier_namespace, inspire_dls:spatial_dataset_identifier_code, inspire_dls:crs and language parameters exactly as specified in the "example" Query elements.

[mwjsanders]

@arantzaetxebarria I have tested this issue in the staging erea with the folowing atom ATOM service: https://service.pdok.nl/kvk/overheidsdiensten/atom/index.xml and this issue seems to be fixed for this ATOM.

However when verifying the returned content type after retrieving the result (for url https://www.nationaalgeoregister.nl/geonetwork/opensearch/dut/download?crs=http%3A%2F%2Fwww.opengis.net%2Fdef%2Fcrs%2FEPSG%2F0%2F28992&language=nl&spatial_dataset_identifier_code=9115b158-de69-48f7-b986-5a6b46b91f72&spatial_dataset_identifier_namespace=http%3A%2F%2Fwww.pdok.nl). The assert is still too strict. The expected content-type in the validator is 'application/atom+xml' , but the server returned 'application/xml;charset=UTF-8' According to the fixes and comments in https://github.com/INSPIRE-MIF/helpdesk-validator/issues/661 this should not be an error, and both content type's should be considered equal.

[Ana-SGA]

Dear all, @fabiovinci, @dperezBM, @arantzaetxebarria,

for this service https://transformiraj.nipp.hr/services/bsp/org.4.8c49ed7b-601e-4c7e-a993-9f1919051ed0/service/atom/id_predefined_dataset we get an error in the INSPIRE Validator for Iterate over Get Spatial Dataset URL. The error pops up in Staging instance too. Can you please take a look - is this the same error as mentioned above? Will this be solved?