koopjs / koop

Transform, query, and download geospatial data on the web.
http://koopjs.github.io
Other
667 stars 128 forks source link

feat: set exceededTransferLimit in metadata #947

Closed rgwozdz closed 6 months ago

rgwozdz commented 7 months ago

Some ArcGIS clients use exceededTransferLimit property on query responses to determine if they should request another page of data. The origin of the value for this field is deep in the Koop dependency chain (Winnow) where the in-memory feature collection is truncated to fit the resultRecordCount parameter. However, pass-through providers may do their own truncation based on resultRecordCount and as a result, never go through Winnow. In such cases, exceededTransferLimit will always have a value of false, even if it should be true. Providers need a way of setting exceededTransferLimit: true when they handle their own truncation.

In this PR, I add the ability for data providers to control the value of exceededTransferLimit by adding it to the geojson metadata.


const geojson = {
  type: 'FeatureCollection',
  features: [ ... ],
  metadata: {
    exceededTransferLimit: true
  }
}

**IMPORTANT:** `exceededTransferLimit` should only be `true` if the query **without limit** returns a record count that is larger that the query with the limit.  Some clients will make a follow-up requests if they receive `exceededTransferLimit: true` - so **it should never be hardcoded to `true` or clients may just keep making the same request**.
changeset-bot[bot] commented 7 months ago

🦋 Changeset detected

Latest commit: 7329e92df67d7eff2e57d8061e9b9c958b239ced

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages | Name | Type | | -------------------------- | ----- | | @koopjs/featureserver | Patch | | @koopjs/koop-core | Minor | | @koopjs/winnow | Major | | @koopjs/output-geoservices | Patch |

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions[bot] commented 7 months ago

Coverage Report (change vs master)

File Path Statements Branches Functions Lines
packages/featureserver/src/route.js 85.71 yellow vs 85.71 yellow 76.47 orange vs 76.47 orange 100 green vs 100 green 85.71 yellow vs 85.71 yellow
packages/featureserver/src/query/render-features.js 100 green vs 80.95 yellow 100 green vs 68 red 100 green vs 100 green 100 green vs 80.95 yellow
packages/winnow/src/normalize-query-options/limit.js 100 green vs 100 green 100 green vs 100 green 100 green vs 100 green 100 green vs 100 green
packages/winnow/src/query/standard-query.js 100 green vs 94.44 yellowGreen 100 green vs 77.77 orange 100 green vs 100 green 100 green vs 94.11 yellowGreen