search

whosonfirst / whosonfirst-www-spelunker

A simple Flask-based spelunker for poking around Who's On First data
BSD 3-Clause "New" or "Revised" License
7 stars 9 forks source link

Pagination is broken #144

Closed henrylyne closed 5 years ago

henrylyne commented 5 years ago

Paging through results just keeps showing the first page of results.

https://spelunker.whosonfirst.org/search/?q=newcastle is the same as https://spelunker.whosonfirst.org/search/?q=newcastle&page=2

thisisaaronland commented 5 years ago

First cut at cursor-based pagination is live. Cursor-based pagination is triggered for queries with results >= 10, 000. Anything less will continue to use pageN style pagination.

Screen Shot 2019-07-16 at 09 48 36 Screen Shot 2019-07-16 at 09 34 06
henrylyne commented 5 years ago

Looks good to me 🚀

thisisaaronland commented 5 years ago

Two steps forward...

WTF {u'status': 500, 'mz:timing': 0.005450010299682617, u'error': {u'failed_shards': [{u'node': u'UiD3rPTuR7yokmTm5_npEg', u'index': u'spelunker_20180213', u'reason': {u'reason': u'Result window is too large, from + size must be less than or equal to: [10000] but was [150000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.', u'type': u'query_phase_execution_exception'}, u'shard': 0}], u'root_cause': [{u'reason': u'Result window is too large, from + size must be less than or equal to: [10000] but was [150000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.', u'type': u'query_phase_execution_exception'}], u'grouped': True, u'reason': u'all shards failed', u'phase': u'query_fetch', u'type': u'search_phase_execution_exception'}}

https://spelunker.whosonfirst.org/placetypes/locality/?page=3000

thisisaaronland commented 5 years ago

Pagination hooks for placetypes. I imagine there are still other queries that need to be accounted for.

https://github.com/whosonfirst/whosonfirst-www-spelunker/commit/7e1f8c9787e247819104d3a93019b066b7030e35

Screen Shot 2019-07-16 at 10 45 55
thisisaaronland commented 5 years ago

Cursor-based pagination has been wired in across the board on the Spelunker.

The Spelunker has also been fully re-indexed (for admin data) as of this morning with the exception of some places in Thailand because of weird ES indexing issues (specifically this place is understood to contain NaN values which... anyway).

Have a poke around and let me know if anything still seems weird or broken.