Currently, when performing a search in the DJDB, every entity matched is
returned and listed. This can cause a timeout if too many entities are matched.
Paging is therefore desirable, whereby only a set amount of items are displayed
at a time, and the user can page forward and backward to see additional items.
However, this is problematic for two reasons, one aesthetic and one technical:
1. The search results are segregated into three categories (Artist, Album, and
Track). How to best incorporate this into a paging scheme?
2. The search system make multiple queries to a collection of search objects
prepared when searchable entities (artists, albums, and tracks) are added to
the datastore. The results of these queries are concatenated and the final list
is sorted. This enables very flexible searches that are not subject to the
limitations of single direct datastore queries (e.g., inequality matching on
only one field, "join"-like behavior). Unfortunately, this means that paging
must be added as a post-processing step, and the entire search must be
completed for every page, which is inefficient and timeouts may still occur.
(Contrast this to a paging scheme [like the one used for browsing the DJDB]
where single queries are made such that only items within current page are
fetched. The search capabilities are limited, but this is efficient and will
never time out.) I'm not sure how to implement paging to get around this
inefficiency and solve the potential for timeouts given the search system we
have. Perhaps somehow use memcache on searches that return a large number of
entities? Still doesn't solve a potential timeout the first time or after the
cache expires or is invalidated by new data.
Any suggestions are very welcome!
Original issue reported on code.google.com by jawax...@gmail.com on 8 Sep 2010 at 3:17
Original issue reported on code.google.com by
jawax...@gmail.comon 8 Sep 2010 at 3:17