search

geopython / pygeoapi

pygeoapi is a Python server implementation of the OGC API suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is open source and released under an MIT license.
https://pygeoapi.io
MIT License
483 stars 258 forks source link

OGR Provider fails to output geojson encoded feature with size bigger than 200MB #371

Closed francbartoli closed 4 years ago

francbartoli commented 4 years ago

Description Working with this dataset:

countries:
        title: Countries
        description: Admin0 official countries from WFP
        keywords:
            - admin0
            - countries
            - WFP
        links:
            - type: text/html
              rel: canonical
              title: information
              href: https://geonode.wfp.org/layers/geonode%3Aglb_bnd_adm0_1
              hreflang: en-EN
        extents:
            spatial:
                bbox: [-180,-90,180,90]
                crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84
            temporal:
                begin: null
                end: now  # or empty
        provider:
            name: OGR
            data:
                source_type: WFS
                source: WFS:https://geonode.wfp.org/geoserver/wfs
                source_srs: EPSG:4326
                target_srs: EPSG:4326
                source_capabilities:
                    paging: True
                source_options:
                    OGR_WFS_LOAD_MULTIPLE_LAYER_DEFN: NO
                open_options:
                    EXPOSE_GML_ID: NO
                    URL: https://geonode.wfp.org/geoserver/wfs?outputformat=json
                gdal_ogr_options:
                    EMPTY_AS_NULL: NO
                    GDAL_CACHEMAX: 64
                    CPL_DEBUG: NO
                    GDAL_HTTP_UNSAFESSL: YES
            id_field: adm0_id
            layer: geonode:glb_bnd_adm0_1

some features that have geojson output bigger than the default OGR_GEOJSON_MAX_OBJ_SIZE=200MB fail with several error messages like:

/pygeoapi/pygeoapi/provider/ogr.py:122} ERROR - Error Number: 1, Type: Failure, Msg: GeoJSON object too complex, please see the OGR_GEOJSON_MAX_OBJ_SIZE environment option

Steps to Reproduce Steps to reproduce the behavior:

For example Indonesia with adm0_id=272

curl -XGET http://localhost:5000/collections/countries/items/272

Expected behavior It is expected the correct geojson payload of the response. This can be mitigated in GDAL version 3.0.2 and above by setting this environment variable:

OGR_GEOJSON_MAX_OBJ_SIZE=500MB

However the error happening in the code is not handled properly

Screenshots/Tracebacks If applicable, add screenshots to help explain your problem.

Environment

Additional context Add any other context about the problem here.

francbartoli commented 4 years ago

However, no exceptions have been thrown when the error happened. This is strange because we have called

gdal.UseExceptions()

in the current code. The result, if the errors are not printed, has all the non-geometrical values null. I think we could mitigate this by throwing an exception if all the values are None and catching it to return a response with a code 500.