search

smnorris / bcdata

Python and command line tools for quick access to DataBC geo-data available via WFS/WCS.
MIT License
29 stars 7 forks source link

catch/report errors due to api service interruptions #186

Open smnorris opened 4 months ago

smnorris commented 4 months ago

See error at https://github.com/smnorris/bcfishpass/actions/runs/9815019608/job/27103503942

Same error with bcdata v0.10.2 and v0.10.4, as long as no cached response is present.

Something must have changed with the response?

smnorris commented 4 months ago

The error occurs when owslib parses the request, it isn't a bcdata function. owslib is pinned to 0.29.*, which has been available for some time - it shouldn't be a change to that that is causing the issue?

smnorris commented 4 months ago

looks to me like it is an error with the response, closing with hope that service is back up next week.

$ curl http://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&request=Getcapabilities

 <?xml version="1.0" encoding="UTF-8"?><ows:ExceptionReport xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="2.0.0" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://openmaps.gov.bc.ca/geo/schemas/ows/1.1.0/owsAll.xsd">
  <ows:Exception exceptionCode="MissingParameterValue" locator="request">
    <ows:ExceptionText>Could not determine geoserver request from http request org.geoserver.monitor.MonitorServletRequest@6cc285b6</ows:ExceptionText>
  </ows:Exception>
</ows:ExceptionReport>
smnorris commented 4 months ago

Also noted here: https://github.com/bcgov/bcdata/issues/339. A similar workaround should be fine for users with known datasets - listing won't work.

smnorris commented 4 months ago

No, bcdata validates against the getcapabilities response, a workaround is not really practical.

smnorris commented 4 months ago

Service is back up!

smnorris commented 4 months ago

More informative error messages are the way to go:

smnorris commented 4 months ago

Service is reported as up on the uptime page but is returning 502s fairly regularly. The retries work for the most part but I suspect depending on WFS during peak fire season is not advisable?

2024-07-22 16:30:15,821:WARNING:bcdata.wfs: HTTP error: 502
2024-07-22 16:30:15,821:WARNING:bcdata.wfs: Response headers: {'Date': 'Mon, 22 Jul 2024 23:29:43 GMT', 'Server': 'Apache', 'Content-Length': '341', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'text/html; charset=iso-8859-1'}
2024-07-22 16:30:15,821:WARNING:bcdata.wfs: Response text: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>