robgaston
commented
9 months ago Related to this, though not well documented, Arches provides data sources on the search map that represent tunable geohash aggregations of the entire search result set. these are similar to, though not exactly, the suggested clustered search results layer without the additional overhead of the described solution. this is not to say that the suggested solution is not appropriate, but simply that we should acknowledge this existing functionality intended to support showing the entire aggregated result set on the map in this discussion.
for reference, Geohash Grid aggregations effectively return cells (whose geographic size can be tuned) with result counts in each cell. this can be useful in conveying density (using heat maps, or bins) but not exact result locations.
chiatt
whatisgalen
mradamcox
The default behavior in the Arches SearchView is to paginate the search request to the appropriate django view,
search.py. While this UX is natural to a paginated UI component like the Search Results component, it presents a fundamental constraint to an inherently unpaginated UI component like the Map Filter. The available UX in the Map Filter component enables a user to either see every single resource instance of a given resource model via the resource layer, a page-limit’s worth of search results from their query, or a combination of both.Not having an unpaginated Map UX is a huge missed opportunity for Arches given that the map filter highlights the first-class status of mappable data, and given the robust power of the combinability of the other search filter components. Why hasn’t it already been done? Well there are technical challenges, of course.
We implemented the desired functionality in HistoricPlacesLA and there were a few core changes to make it run in a timely manner:
Enable Backend Caching of Search results For the largest queries by result volume, many queries were repeated by a lot of the users. For “Anonymous” users in particular, given that they all had identical data permissions, it made sense to cache one set for all of them. The cache key would have to capture the whole query and also include some snapshot of the database in case search results had been mutated. We used the count from the EditLog model. Determining when these caches timeout (outside of the snapshot being embedded in the cache key) is a matter of preference and should be its own setting at the project level. Without embedding backend caching, the demand on Elasticsearch could increase significantly given the desired load (see next point).
Enable Unpaginated Search results To paint all the features on the map, clustered or in whatever layer you style, a search query would need to request all search results instead of a single page’s worth. This means the default query in SearchView should be able to interrogate the instance settings for whether to paginate every search query. This means the search view should be able to handle the case of no paging-filter present on the query object.
Enable Frontend & Backend Caching of Search Result Geometries Because the FeatureCollections of points and polygons tend to eat up compute during JSON conversion, there’s a lot of utility in caching a “master” lookup of geometry to lighten the payload of any subsequent query’s search results. In HistoricPlacesLA, a compressed geometry lookup for 50,000 mixed point/polygons can be cached to the browser at about 3mb. There is a hard limit of 5mb since browsers don’t support larger browser caches and a real tradeoff to consider regarding whether it’s better to frontload first to save time later vs waiting a few seconds longer for those larger geom-heavy search results. This should be its own optional flow to be configured by the implementer, but it is a useful pattern that merits being available rather than reinvented.
Create a Map Layer to cluster Search results The behavior of a dynamic search map differs from both the search-result layer and the resource-layer stylings. A functional map layer for unpaginated search results should by default have clustering behavior similar to the resource-layer. There isn’t a compelling reason not to enable UI-supported styling for this overlay map-layer, but in the near term it could be styled directly in the json similar to any other overlay.
Example Implementation: HPLA HistoricPlacesLA has over 50,000 resources with geometries. For HPLA, our overall strategy was to cache both: A) a “master” set of search result geometries on the backend and B) literally every search result set. When a user lands on the home page (index.html), the page would request the cached geometry lookup from the backend via search_results and then compress and cache this set in the client’s browser localStorage cache. Typical wait time for this to complete was 3-8 seconds. Half of the runtime goes to compression of the geom lookup.
When the user proceeds to Search, their search query returns an unpaginated result set sans any point or polygons (best case from the backend search result cache). On the frontend, each search result gets compared against the frontend geom cache and if matched, gets “hydrated” with a cached geom. The ligher payload (even when no cached results exist) makes a noticeable difference in load time.
The end product is a dynamic, unpaginated map query that does what you expect it to. While the default functionality in SearchView runs in less than 1 second, for any project that has even half of the geoms HPLA has, the payoff of a dynamic map will likely merit the extra 2-3 seconds spent waiting.
I think there’s a strong case to be made that this functionality deserves to be “opt-in” in core Arches.