search

ZGIS / smart-csw-ingester

CSW to Lucene Index
Apache License 2.0
0 stars 1 forks source link

Koordinates CSW XML Query review #39

Closed allixender closed 6 years ago

allixender commented 6 years ago

Suddenly, the queries aren't processed properly anymore:

[info] s.LuceneIndexBuilderActor - Querying mfe on url https://data.mfe.govt.nz/services/csw/
[warn] s.LuceneIndexBuilderActor - Got XML Exception Response. Text: Exception: document not well-formed.
Error: None.

[info] s.LuceneIndexBuilderActor - Querying linz on url https://data.linz.govt.nz/services/csw/
[warn] s.LuceneIndexBuilderActor - Got XML Exception Response. Text: Exception: document not well-formed.
Error: None.

[info] s.LuceneIndexBuilderActor - Querying landcare on url https://lris.scinfo.org.nz/services/csw/
[warn] s.LuceneIndexBuilderActor - Got XML Exception Response. Text: Exception: document not well-formed.
Error: None.

Manual test via POST shows following full response:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- pycsw 1.10.5.dev20170420050656 -->
<ows:ExceptionReport 
    xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" 
    xmlns:atom="http://www.w3.org/2005/Atom" 
    xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" 
    xmlns:dc="http://purl.org/dc/elements/1.1/" 
    xmlns:dct="http://purl.org/dc/terms/" 
    xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" 
    xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" 
    xmlns:gco="http://www.isotc211.org/2005/gco" 
    xmlns:gmd="http://www.isotc211.org/2005/gmd" 
    xmlns:gml="http://www.opengis.net/gml" 
    xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" 
    xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" 
    xmlns:ogc="http://www.opengis.net/ogc" 
    xmlns:os="http://a9.com/-/spec/opensearch/1.1/" 
    xmlns:ows="http://www.opengis.net/ows" 
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
    xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" 
    xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" 
    xmlns:srv="http://www.isotc211.org/2005/srv" 
    xmlns:xlink="http://www.w3.org/1999/xlink" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" language="en-GB" version="1.2.0" xsi:schemaLocation="http://www.opengis.net/ows http://schemas.opengis.net/ows/1.0.0/owsExceptionReport.xsd">
    <ows:Exception exceptionCode="NoApplicableCode" locator="service">
        <ows:ExceptionText>Exception: document not well-formed.
Error: None.</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>
tomkralidis commented 6 years ago

We're having similar issues in QGIS (see https://issues.qgis.org/issues/18893). From the XML response (pycsw version) it looks like a fork of pycsw with a modified version.

allixender commented 6 years ago

@tomkralidis Yes, we can confirm. Interesting to know, that it seems to pop-up somewhere else, too. Anyone a contact there to inquire. Because it worked for years more or less flawlessly. And now on several of the instances of the Koordinates Data Portal the CSW endpoint behaves like that.

tomkralidis commented 6 years ago

cc'ing @rcoup here for any potential updates.

rcoup commented 6 years ago

👋 all, I'm from Koordinates.

@tomkralidis correct, fundamentally it's PyCSW doing most of the CSW protocol work, but against a different backend datastore.

Suddenly, the queries aren't processed properly anymore:

@allixender got any idea when this started (roughly)? I've had a quick review of our changelog and nothing's jumping out at me in terms of relevant changes recently, and our test suite is passing. But I'll talk to the team and get back to you.

Manual test via POST shows following full response:

Possible to get a copy of your test queries in a gist?

tomkralidis commented 6 years ago

@rcoup aside: any info on the different backend? Feel free to email me offline.

allixender commented 6 years ago

@rcoup Following XML query for the 3 services for example:

https://data.linz.govt.nz/services/csw/

https://data.mfe.govt.nz/services/csw/

https://lris.scinfo.org.nz/services/csw/

<GetRecords xmlns="http://www.opengis.net/cat/csw/2.0.2"
xmlns:ogc="http://www.opengis.net/ogc"
xmlns:ows="http://www.opengis.net/ows"
xmlns:gml="http://www.opengis.net/gml"
xmlns:gmd="http://www.isotc211.org/2005/gmd"

service="CSW"
version="2.0.2"
maxRecords="3"
startPosition="1"
resultType="results"
outputFormat="application/xml"
outputSchema="http://www.opengis.net/cat/csw/2.0.2">
<Query typeNames="gmd:MD_Metadata">
<ElementSetName>brief</ElementSetName>
<Constraint version="1.1.0">
  <ogc:Filter>
<ogc:Intersects>
<ogc:PropertyName>ows:BoundingBox</ogc:PropertyName>
<gml:Envelope srsName="http://www.open-gis.net/def/crs/epsg/0/4326">
<gml:lowerCorner>14.05 46.46</gml:lowerCorner>
<gml:upperCorner>17.24 48.42</gml:upperCorner>
</gml:Envelope>
</ogc:Intersects>
</ogc:Filter>
</Constraint>
</Query>
</GetRecords>

returns:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- pycsw 1.10.5.dev20170420050656 -->
<ows:ExceptionReport 
    xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" 
    xmlns:atom="http://www.w3.org/2005/Atom" 
    xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" 
    xmlns:dc="http://purl.org/dc/elements/1.1/" 
    xmlns:dct="http://purl.org/dc/terms/" 
    xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" 
    xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" 
    xmlns:gco="http://www.isotc211.org/2005/gco" 
    xmlns:gmd="http://www.isotc211.org/2005/gmd" 
    xmlns:gml="http://www.opengis.net/gml" 
    xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" 
    xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" 
    xmlns:ogc="http://www.opengis.net/ogc" 
    xmlns:os="http://a9.com/-/spec/opensearch/1.1/" 
    xmlns:ows="http://www.opengis.net/ows" 
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
    xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" 
    xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" 
    xmlns:srv="http://www.isotc211.org/2005/srv" 
    xmlns:xlink="http://www.w3.org/1999/xlink" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" language="en-GB" version="1.2.0" xsi:schemaLocation="http://www.opengis.net/ows http://schemas.opengis.net/ows/1.0.0/owsExceptionReport.xsd">
    <ows:Exception exceptionCode="NoApplicableCode" locator="service">
        <ows:ExceptionText>Exception: document not well-formed.
Error: None.</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

We opened that issue on May 24, where this effect could only have been happening for a few weeks max I believe.

Hope this info helps.

rcoup commented 6 years ago

@allixender we made a change and AFAICT this is resolved now. Are you able to check that you're seeing what you expect? Will be the same across all the Koordinates-hosted sites.

@tomkralidis I'll drop you an email

allixender commented 6 years ago

@rcoup @tomkralidis Whoop, the requests are successful again, outstanding work 👍 😃 We'll observe in production for a while, and then I guess we can close this

Also https://issues.qgis.org/issues/18893 MetaSearch plugin in QGIS finds LINZ data again.

rcoup commented 6 years ago

@allixender great! Get us on support@koordinates.com if you ever need any help in future with our sites from a developer/technical point of view.