search

slatedb / slatedb

A cloud native embedded storage engine built on object storage.
https://slatedb.io
Apache License 2.0
926 stars 46 forks source link

local sst cache eviction #159

Open flaneur2020 opened 3 weeks ago

flaneur2020 commented 3 weeks ago

a follow up of https://github.com/slatedb/slatedb/issues/9

came across this strategy might be helpful while dealing with millions of cache files:

Cache size (in MiB, default to 102400) and minimum ratio of free size (default 0.1). Both parameters is able to control cache size, if any of the two criteria is met, client will evict cache using the simple 2-random strategy, i.e. randomly pick 2 blocks, and drop the one with an earlier access time. Under most real world scenarios, the overall outcome is similar to LRU algorithm, while random-2 comes with lower overhead.

from https://juicefs.com/docs/cloud/guide/cache/#data-cache

flaneur2020 commented 3 weeks ago

and please assign this issue to me, i'll continue work on it after a no-auto-evict local cache in #139 is usable : )

once #139 is usable, it'd be nice to have some benchmarks in parallel with the implementation of eviction

criccomini commented 3 weeks ago

Awesome! Assigned to you.

@rodesai is in the process of merging a benchmark tool in #148. Think this would be enough to run some benchmarks if you instrument the code with metrics (#145 just got merged)?