search

smnorris / bcdata

Python and command line tools for quick access to DataBC geo-data available via WFS/WCS.
MIT License
29 stars 7 forks source link

owslib getcapabilities fails #141

Closed smnorris closed 1 year ago

smnorris commented 1 year ago

As url currently defined (and has been for some time), owslib is failing, no longer finding OperationsMetadata in the response:

>>> from owslib.wfs import WebFeatureService
>>> WebFeatureService("http://openmaps.gov.bc.ca/geo/ows", version="2.0.0")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/snorris/Projects/repo/bcdata/bcdata_env/lib/python3.9/site-packages/owslib/wfs.py", line 56, in WebFeatureService
    return wfs200.WebFeatureService_2_0_0(
  File "/Users/snorris/Projects/repo/bcdata/bcdata_env/lib/python3.9/site-packages/owslib/feature/wfs200.py", line 77, in __new__
    obj.__init__(
  File "/Users/snorris/Projects/repo/bcdata/bcdata_env/lib/python3.9/site-packages/owslib/feature/wfs200.py", line 129, in __init__
    self._buildMetadata(parse_remote_metadata)
  File "/Users/snorris/Projects/repo/bcdata/bcdata_env/lib/python3.9/site-packages/owslib/feature/wfs200.py", line 164, in _buildMetadata
    for elem in self._capabilities.find(nspath("OperationsMetadata"))[:]:
TypeError: 'NoneType' object is not subscriptable

The data does not show up with a direct request either:

curl -o capabilities.xml "https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&version=2.0.0&request=GetCapabilities" 
cat capabilities.xml | grep OperationsMetadata

Looks to me like WFS getCapabilities response is now a WMS styling doc, the schema info is no longer present either, breaking bcdata.

I can alert DataBC but we could switch to using DescribeFeatureType to get the schema and geometry name.

smnorris commented 1 year ago

for example, https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&version=2.0.0&request=DescribeFeatureType&typeNames=WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW

<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:pub="http://delivery.openmaps.gov.bc.ca/geo/" xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://delivery.openmaps.gov.bc.ca/geo/">
  <xsd:import namespace="http://www.opengis.net/gml/3.2" schemaLocation="http://openmaps.gov.bc.ca/geo/schemas/gml/3.2.1/gml.xsd"/>
  <xsd:complexType name="WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVWType">
    <xsd:complexContent>
      <xsd:extension base="gml:AbstractFeatureType">
        <xsd:sequence>
          <xsd:element maxOccurs="1" minOccurs="1" name="CUSTODIAN_ORG_DESCRIPTION" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="BUSINESS_CATEGORY_CLASS" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="BUSINESS_CATEGORY_DESCRIPTION" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="OCCUPANT_TYPE_DESCRIPTION" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="SOURCE_DATA_ID" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="SUPPLIED_SOURCE_ID_IND" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="AIRPORT_NAME" nillable="false" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="DESCRIPTION" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="PHYSICAL_ADDRESS" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="ALIAS_ADDRESS" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="STREET_ADDRESS" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="POSTAL_CODE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="LOCALITY" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="CONTACT_PHONE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="CONTACT_EMAIL" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="CONTACT_FAX" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="WEBSITE_URL" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="IMAGE_URL" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="LATITUDE" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="LONGITUDE" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="KEYWORDS" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="DATE_UPDATED" nillable="true" type="xsd:date"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SITE_GEOCODED_IND" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="AERODROME_STATUS" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="AIRCRAFT_ACCESS_IND" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="DATA_SOURCE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="DATA_SOURCE_YEAR" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="ELEVATION" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="FUEL_AVAILABILITY_IND" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="HELICOPTER_ACCESS_IND" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="IATA_CODE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="ICAO_CODE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="MAX_RUNWAY_LENGTH" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="NUMBER_OF_RUNWAYS" nillable="true" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="OIL_AVAILABILITY_IND" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="RUNWAY_SURFACE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SEAPLANE_ACCESS_IND" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="TC_LID_CODE" nillable="true" type="xsd:string"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SHAPE" nillable="true" type="gml:GeometryPropertyType"/>
          <xsd:element maxOccurs="1" minOccurs="1" name="SEQUENCE_ID" nillable="false" type="xsd:decimal"/>
          <xsd:element maxOccurs="1" minOccurs="0" name="SE_ANNO_CAD_DATA" nillable="true" type="xsd:hexBinary"/>
        </xsd:sequence>
      </xsd:extension>
    </xsd:complexContent>
  </xsd:complexType>
  <xsd:element name="WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW" substitutionGroup="gml:AbstractFeature" type="pub:WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVWType"/>
</xsd:schema>
smnorris commented 1 year ago

or, a tidy schema via owslib:

from owslib.feature import schema
schema = schema.get_schema(
    "https://openmaps.gov.bc.ca/geo/pub/ows", 
    typename="WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW", version="2.0.0"
)

{'properties': {'CUSTODIAN_ORG_DESCRIPTION': 'string',
...
smnorris commented 1 year ago

Listing the available layers is the only missing piece required from GetCapabilities. An unqualified DescribeFeatureType should work to list all layers, but the server does not like that:

curl "https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&version=2.0.0&request=DescribeFeatureType"
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

Perhaps working through the catalogue api could give a list of layers?

smnorris commented 1 year ago

Added a static list of layers to the module as a workaround. List came from cached capabilities.xml. Closing this for now as we don't strictly need a getcapabilites response, just some way of listing the available datasets.