search

geopython / pygeoapi

pygeoapi is a Python server implementation of the OGC API suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is open source and released under an MIT license.
https://pygeoapi.io
MIT License
490 stars 260 forks source link

Pagination issue in v0.16 #1594

Closed geoportallux closed 5 months ago

geoportallux commented 6 months ago

Description We have a regression with pagination on a layer using a PostGIS Point Layer, the "next" link disappears before having cycled through every record

Steps to Reproduce

This is a request on a v15 server:

https://ogc-api-poc.geoportail.lu/collections/adresses_pg/items?offset=3000&limit=100&bbox=6.06309026297383014,49.61285059172146816,6.13628918983878346,49.62513645744658675&f=json

It contains a "next" link to request the next set of 100 items

Here is the same request on a v 0.16.0 server

https://ogc-api-poc-v16.geoportail.lu/collections/adresses_pg/items?offset=3000&limit=100&bbox=6.06309026297383014,49.61285059172146816,6.13628918983878346,49.62513645744658675&f=json

There is no "next" link in this case

If I put the source in QGIS, the v 0.16.0 does not show all the features, whereas it does in v 0.15.0

This is my config in both cases:

adresses_pg:
    links:  # list of 1..n related links
        - type: text/csv  # MIME type
          rel: canonical  # link relations per https://www.iana.org/assignments/link-relations/link-relations.xhtml
          title: data  # title
          href: https://github.com/mapserver/mapserver/blob/branch-7-0/msautotest/wxs/data/obs.csv  # URL
          hreflang: en-US  # language
    type: collection
    title: adresses_from_pg
    description: adresses directly from pg
    keywords:
        - adresses
    extents:
        spatial:
            bbox: [5.73,49.44,6.53,50.19]
            crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84
    providers:
        - type: feature
          name: PostgreSQL
          data:
             REMOVED
          id_field: ogcid
          table: adresses
          geom_field: geom

Expected behavior For both versions, all the layers should be shown in QGIS when panning through the map

Screenshots/Tracebacks image

Environment

Additional context Add any other context about the problem here.

totycro commented 6 months ago

Here is the crucial line for this behavior: https://github.com/geopython/pygeoapi/blob/master/pygeoapi/api.py#1920

So it checks if there are enough numberMatched to get a next page. In this example, numberMatched is 1926.

The postgres provider seems to only count the values after the offset, so in total, this query gives 4926 matches.

However the api assumes that numberMatched is the total number of matches.

So to fix this, we either need to change the behavior of the postgres provider or the API. Also, we might need to describe what numberMatched really means (either total number of matches, or number of matches after offset).

Btw the previous version did contain the inverse bug, i.e. having a next page where there shouldn't be one: https://github.com/geopython/pygeoapi/issues/1237

tomkralidis commented 6 months ago

Given that numberMatched means the total number of matches, this sounds like an update needed in the provider.