Caching

[autoencoder]

Discussion on query-caching plans:

Note: Caching of query sessions is valuable for pagination & useful URLs on search results, good idea for us to use Redis for that.

Query caching turns out to be... quite unique. Depending on the implementation, possibly even degrading performance. Different from typical web app caching, or even typical database caching. Considerations:

• Vastly flatter power-law curve for queries than, e.g. the power law curve of web site views per page on a typical website. > 50% of queries only seen once ever. As technology progresses and personalization (user-based, session-based, other context based) increases, this will near 100% of queries being unique, when viewed from the top level of the stack.
• User satisfaction scores for search tend to be extremely dominated by worst-case response times (e.g. worst 5%) instead of average-case response times. With possibly 50%+ of queries being unique, top-level query-caching doesn't do anything for 50% of queries, let alone the worst 5%.
• It's so important for search index shards to each fit entirely in RAM, and each shard having as much RAM as possible, that reserving significant RAM for a query cache instead of giving that RAM to the index can have real negative overall performance impact.

So adding top-level query caching is still useful, but unlike for regular web apps where top-level caching alone is practically all you need, here it only covers a small fraction of what's needed. Most of the caching will occur at deeper levels of the search stack.

Most of the Indexing components have built-in caching mechanisms for this, and other reasons. Investigating those and enabling some of them as we go.

[parkan]

Presumably top-level caching is still important for soaking up load/spikes at popular queries. Down to discuss domain specific caching strategies.

[autoencoder]

Closing in favor of new storage system & ANN index design.