search

geopython / pywps

PyWPS is an implementation of the Web Processing Service standard from the Open Geospatial Consortium. PyWPS is written in Python.
https://pywps.org
MIT License
175 stars 117 forks source link

BoundingBox input types cannot be serialised to submit to Slurm cluster #610

Closed agstephens closed 2 years ago

agstephens commented 3 years ago

Description

When sending a Bounding Box input type to the WPS server, it fails with an error:

<!-- PyWPS 4.4.0 -->
<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd" version="1.0.0">
  <ows:Exception exceptionCode="SchedulerNotAvailable" locator="" >
      <ows:ExceptionText>Could not submit job: Object of type Crs is not JSON serializable</ows:ExceptionText>
  </ows:Exception>
</ows:ExceptionReport>

This seems to happen when trying to submit jobs to a Slurm cluster - so serialisation is required before submitting to Slurm, and it cannot do that.

Environment

Steps to Reproduce

I am connecting from a GUI that posts the request, not sure how to reproduce.

Additional Information

I was able to fix if I make these two hacks to owslib and pywps:

owslib/ows.py
=========

    238 class BoundingBox(object):
    239     """Initialize an OWS BoundingBox construct"""
    240     def __init__(self, elem, namespace=DEFAULT_OWS_NAMESPACE):
    241         self.minx = None
    242         self.miny = None
    243         self.maxx = None
    244         self.maxy = None
    245         self.crs = None
    246         self.dimensions = 2
    247         if elem is None:
    248             return
    249         val = elem.attrib.get('crs') or elem.attrib.get('{{{}}}crs'.format(namespace))
    250         if val:
    251             try:
ADDED LINE:    252                 val = 'urn:ogc:def:crs:OGC:2:84'
    253                 self.crs = crs.Crs(val)
    254             except (AttributeError, ValueError):
    255                 LOGGER.warning('Invalid CRS %r. Expected integer' % val)

pywps/app/WPSRequest.py
==================

    444         # Using OWSlib BoundingBox
    445         from owslib.ows import BoundingBox
    446         bbox_datas = xpath_ns(input_el, './wps:Data/wps:BoundingBoxData')
    447         if bbox_datas:
    448             for bbox_data in bbox_datas:
    449                 bbox = BoundingBox(bbox_data)
    450                 LOGGER.debug("parse bbox: minx={}, miny={}, maxx={},maxy={}".format(
    451                     bbox.minx, bbox.miny, bbox.maxx, bbox.maxy))
    452                 inpt = {}
    453                 inpt['identifier'] = identifier_el.text
    454                 inpt['data'] = [bbox.minx, bbox.miny, bbox.maxx, bbox.maxy]
CHANGED LINE:    455                 inpt['crs'] = '' #bbox.crs
    456                 inpt['dimensions'] = bbox.dimensions
    457                 the_inputs[identifier].append(inpt)
    458     return the_inputs

I don't really know what these hacked lines do but it gets things working.

I'm happy to provide more info and test when I have more time.

geotom commented 2 years ago

I have the same problem now: First I had a general issue with getting the BBOX parameter work, but this was to my own mistake. But now I have the same problem with the CRS not being serializable.

It happens very randomly: I can submit a job with the following input:

<wps:Input>
    <ows:Identifier>area_of_interest</ows:Identifier>
    <wps:Data>
        <wps:BoundingBoxData crs="epsg:4326" dimensions="2">
            <ows:LowerCorner>-90.0 -45.0</ows:LowerCorner>
            <ows:UpperCorner>90.0 45.0</ows:UpperCorner>
        </wps:BoundingBoxData>
    </wps:Data>
</wps:Input>

It gets accepted and runs. After a few more executions I suddenly get this answer:

<?xml version="1.0" encoding="UTF-8"?>
<!-- PyWPS 4.5.1 -->
<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd" version="1.0.0">
    <ows:Exception exceptionCode="NoApplicableCode" locator="" >
        <ows:ExceptionText>No applicable error code, please check error log.</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

and the log file tells me:

2022-03-21 08:57:37,137] [ERROR] file=/usr/local/lib/python3.8/dist-packages/pywps/exceptions.py line=48 module=exceptions function=__init__ Exception: code: 500, description: No applicable error code, please check error log., locator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/Service.py", line 314, in call
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/Service.py", line 304, in call
    response = self.execute(
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/Service.py", line 82, in execute
    return self._parse_and_execute(process, wps_request, uuid)
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/Service.py", line 145, in _parse_and_execute
    wps_response = process.execute(wps_request, uuid)
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/Process.py", line 149, in execute
    wps_response = self._execute_process(self.async_, wps_request, wps_response)
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/Process.py", line 215, in _execute_process
    dblog.store_process(self.uuid, wps_request)
  File "/usr/local/lib/python3.8/dist-packages/pywps/dblog.py", line 192, in store_process
    request_json = request.json
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/WPSRequest.py", line 461, in json
    return json.dumps(obj, allow_nan=False, cls=ExtendedJSONEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/dist-packages/pywps/app/WPSRequest.py", line 442, in default
    encoded_object = json.JSONEncoder.default(self, obj)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

To make it execute again or at least not get an error is to remove the CRS from the BBox input like this:

<wps:Input>
    <ows:Identifier>area_of_interest</ows:Identifier>
    <wps:Data>
        <wps:BoundingBoxData dimensions="2">
            <ows:LowerCorner>-90.0 -45.0</ows:LowerCorner>
            <ows:UpperCorner>90.0 45.0</ows:UpperCorner>
        </wps:BoundingBoxData>
    </wps:Data>
</wps:Input>

I am calling the service asynchronously and have pyWPS integrated into a flask app that is run via gunicorn and exposed via nginx. The same as described in the Howtos for production system. I suspect it has to do if jobs are immediately executed by a worker or put into a local queue. It is all very sporadic and my feeling is that increasing the number of workers helped, but the whole pyWPS interface degrades over a time

geotom commented 2 years ago

Could it be a problem with the BBox input serialisation if it needs to be scheduled via a queue?

geotom commented 2 years ago

My further investigations The object that is not JSON encodable is urn:ogc:def:crs:EPSG::4326.

It fails at: /pywps/app/WPSRequest.py", line 444, in default

cehbrecht commented 2 years ago

fixed by PR #650