search

bird-house / flyingpigeon

WPS processes for climate model data, indices and extreme events
http://flyingpigeon.readthedocs.io/en/latest/
Apache License 2.0
19 stars 15 forks source link

WFS feature request length limit? #290

Closed Zeitsperre closed 3 years ago

Zeitsperre commented 5 years ago

This is a follow-up on the some of the strange behaviour I noted in #287.

Presently, WFS feature requests using FlyingPigeon, specifically wps_subset_wfs_polygon.py, seem to be working for the most part. The issue I'm currently dealing with now has to do with the built-in function get_feature from subset_base.py https://github.com/bird-house/flyingpigeon/blob/56b5f4e9d4b8fa9354fe8151f58d7596762a07ea/flyingpigeon/processes/subset_base.py#L50

The issue I've found is that If I send a WFS request with more than a few hundred features requested (>~220), the request fails abruptly and without much reasoning. Exception handling in Flyingpigeon is clear enough to show that my request is seen and well-parsed:

Traceback (most recent call last):
  File "/home/tjs/git/flyingpigeon/src/pywps/pywps/app/Process.py", line 208, in _run_process
    self.handler(wps_request, wps_response)  # the user must update the wps_response.
  File "/home/tjs/git/flyingpigeon/flyingpigeon/processes/wps_subset_wfs_polygon.py", line 65, in _handler
    geoms = self.parse_feature(request)
  File "/home/tjs/git/flyingpigeon/flyingpigeon/processes/subset_base.py", line 125, in parse_feature
    raise Exception(msg)
Exception: Failed to fetch features.
geoserver: https://boreas.ouranos.ca/geoserver/wfs 
typename: public:USGS_HydroBASINS_lake_na_lev12
features ['USGS_HydroBASINS_lake_na_lev12.67061', 
    ... ... ...
'USGS_HydroBASINS_lake_na_lev12.67424']
Expecting value: line 1 column 1 (char 0)

Following the stack down, I believe the error may be due to a limit set in the OWSLib WFS request parser:

https://github.com/geopython/OWSLib/blob/3338340e6a9c19dd3388240815d35d60a0d0cf4c/owslib/feature/wfs100.py#L191

My theory is that there's a limit to how long a request can be before OWSLib throws it out. Is this a possible explanation?

huard commented 5 years ago

GeoServer might have limits on GET requests as well. One possible solution is to use POST requests, which usually have higher size limits. With WFS2.0.0, getfeature has a method parameter that you can set to "Post". https://github.com/geopython/OWSLib/blob/3338340e6a9c19dd3388240815d35d60a0d0cf4c/owslib/feature/wfs200.py#L225 Not clear however if it is really implemented.

huard commented 5 years ago

See https://github.com/geopython/OWSLib/issues/439