bcdata cli failing on specific data bc layer: ground-water-wells

[haileye]

Hi,

I have a working version of bcdata (works on bc-airports & ground-water-aquifers) and it used to work on ground-water-wells but it stopped working on Jan. 15, 2020. The last time the data was modified by Data BC was Jan. 15, 2020 so I imagine it is related. I saw an alert from the data custodian saying this data was going to modified.

The specific layer is ground-water-wells

It appears in the API catalogue csv And I can download it manually from Data BC.

The error I'm seeing when I run either CLI call bcdata dump ground-water-wells or bcdata bc2pg ground-water-wells ... is:

~$ bcdata dump ground-water-wells Traceback (most recent call last): File "/opt/venvs/bcdata/bin/bcdata", line 10, in <module> sys.exit(cli()) File "/opt/venvs/bcdata/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/opt/venvs/bcdata/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/opt/venvs/bcdata/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/venvs/bcdata/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/venvs/bcdata/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/opt/venvs/bcdata/lib/python3.6/site-packages/bcdata/cli.py", line 186, in dump data = bcdata.get_data(table, query=query, bounds=bounds, bounds_crs=bounds_crs) File "/opt/venvs/bcdata/lib/python3.6/site-packages/bcdata/wfs.py", line 214, in get_data pagesize=pagesize, File "/opt/venvs/bcdata/lib/python3.6/site-packages/bcdata/wfs.py", line 147, in define_request n = bcdata.get_count(table, query=query) File "/opt/venvs/bcdata/lib/python3.6/site-packages/bcdata/wfs.py", line 119, in get_count return int(ET.fromstring(r.text).attrib["numberMatched"]) KeyError: 'numberMatched'

Thanks for you attention! Cheers, Hailey

[smnorris]

same issue here:

$ bcdata info -vv WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_WRBC_SVW
DEBUG:owslib:building WFS http://openmaps.gov.bc.ca/geo/ows
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): openmaps.gov.bc.ca:80
DEBUG:urllib3.connectionpool:http://openmaps.gov.bc.ca:80 "GET /geo/ows?service=WFS&request=GetCapabilities&version=2.0.0 HTTP/1.1" 200 None
DEBUG:owslib:building WFS http://openmaps.gov.bc.ca/geo/ows
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): openmaps.gov.bc.ca:80
DEBUG:urllib3.connectionpool:http://openmaps.gov.bc.ca:80 "GET /geo/ows?service=WFS&request=GetCapabilities&version=2.0.0 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openmaps.gov.bc.ca:443
DEBUG:urllib3.connectionpool:https://openmaps.gov.bc.ca:443 "GET /geo/pub/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_WRBC_SVW&resultType=hits&outputFormat=json HTTP/1.1" 400 None
Traceback (most recent call last):
  File "/usr/local/bin/bcdata", line 11, in <module>
    load_entry_point('bcdata', 'console_scripts', 'bcdata')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/snorris/Projects/geobc/bcdata/bcdata/cli.py", line 136, in info
    info["count"] = bcdata.get_count(table)
  File "/Users/snorris/Projects/geobc/bcdata/bcdata/wfs.py", line 133, in get_count
    return int(ET.fromstring(r.text).attrib["numberMatched"])
KeyError: 'numberMatched'

Because you mentioned that the record has changed, I tried refreshing the local cache of layer names and re-running - the error is then a bit more informative:

$ bcdata list -r
...
$ bcdata info -vv WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_WRBC_SVW
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.data.gov.bc.ca:443
DEBUG:urllib3.connectionpool:https://catalogue.data.gov.bc.ca:443 "GET /api/3/action/package_show?id=WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_WRBC_SVW HTTP/1.1" 404 146
Traceback (most recent call last):
  File "/usr/local/bin/bcdata", line 11, in <module>
    load_entry_point('bcdata', 'console_scripts', 'bcdata')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/snorris/Projects/geobc/bcdata/bcdata/cli.py", line 132, in info
    table = bcdata.validate_name(dataset)
  File "/Users/snorris/Projects/geobc/bcdata/bcdata/wfs.py", line 90, in validate_name
    return get_table_name(dataset)
  File "/Users/snorris/Projects/geobc/bcdata/bcdata/wfs.py", line 67, in get_table_name
    raise ValueError("{d} is not present in DataBC API list".format(d=package))
ValueError: WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_WRBC_SVW is not present in DataBC API list

And, looking more closely at what the WFS is providing from that schema:

$ bcdata list | grep WHSE_WATER_MANAGEMENT.GW_
WHSE_WATER_MANAGEMENT.GW_AQFR_VULN_SWI_PUMP_COAST_SP
WHSE_WATER_MANAGEMENT.GW_AQFR_VULN_SWI_PUMP_SP
WHSE_WATER_MANAGEMENT.GW_AQUIFERS_CLASSIFICATION_SVW
WHSE_WATER_MANAGEMENT.GW_AQUIFER_VLNRBLTY_RPT_IDX_SP
WHSE_WATER_MANAGEMENT.GW_AQUIFER_VULNERABILITY_POLY
WHSE_WATER_MANAGEMENT.GW_CHLORIDE_POLY
WHSE_WATER_MANAGEMENT.GW_GWQ_IMPACT_NPS_POLY
WHSE_WATER_MANAGEMENT.GW_HARDNESS_POLY
WHSE_WATER_MANAGEMENT.GW_NAQUADAT_STATION_SP
WHSE_WATER_MANAGEMENT.GW_NITRATE_NITROGEN_POLY
WHSE_WATER_MANAGEMENT.GW_SPECIFIC_CONDUCTANCE_POLY
WHSE_WATER_MANAGEMENT.GW_TOTAL_ALKALINITY_POLY
WHSE_WATER_MANAGEMENT.GW_TOTAL_DISSOLVED_SOLID_POLY
WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_LITHOLOGY_SP
WHSE_WATER_MANAGEMENT.GW_WELL_CAPTURE_ZONES_SP

[wburt]

The data models for three groundwater datasets in the BCGW were updated mid-january: The WMS service is broken. I know its on the fix list. https://openmaps.gov.bc.ca/geo/pub/WHSE_WATER_MANAGEMENT.GW_WATER_WELLS_WRBC_SVW/ows?service=WMS&request=GetCapabilities

[smnorris]

Thanks Will. I might tweak how bcdata uses its cache of layer names so that the "not present" error shows up in cases like this.

[haileye]

I'll wait until the WMS service is fixed, thank you Simon and Will!