search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.39k stars 3.68k forks source link

Caffeine-based remote cache with adaptive admission windows #7186

Open leventov opened 5 years ago

leventov commented 5 years ago

Caffeine 2.7.0 introduces a cool feature of adaptive admission window size to balance between recency and frequency.

This issue is about the idea to apply this in Druid and have admission windows on Historical nodes to "prove" that segment results are "worth" to be propagated to remote dedicated cache nodes (be it memcached or Java servers running Caffeine).

This issue exists to track https://github.com/ben-manes/caffeine/issues/303, where I think the main discussion should happen.

leventov commented 5 years ago

Segments are served by multiple historicals, and although currently Brokers almost never take advantage of that, there are multiple reasons why they should start query different historicals more often (identified, for example, in this post, and also mentioned in the issue about online aggregation). Thus the same segment might have separate be tracked in several independent admission windows on different historicals. If this is a big obstacle for the idea and how it can be solved (randomization?) is yet to be determined.