search

opendata / Open-Data-Needs

An ongoing effort to catalog the holes in the open data ecosystem. [RETIRED]
15 stars 0 forks source link

Retrieving data ranges #9

Open waldoj opened 9 years ago

waldoj commented 9 years ago

Sometimes you only need a narrow slice of a large dataset, but getting it necessitates transferring the entire dataset, and then extracting what's needed. This can be taxing (or even impossible) due to bandwidth and memory restrictions. Of course, some data providers anticipate this, and provide downloads of subsets of their datasets, but it's impractical to provide all possible slices of data that users might need.

We need a method of retrieving only selected portions of large datasets. I expect that Dat will be a solution to this, but of course more solutions are always good.

This was an issue raised by a SRCCON attendee in July, at a discussion about open data needs.

ajturner commented 9 years ago

FWIW the 'GeoServices' API has a /query that provides SQL like filtering for subsets. see the feature/layer docs.

Koop is designed as an open-source wrapper for any API or data source to provide a similar API with subset and format conversion.

Beyond just the data you can also get statistics and roll up aggregations.