LRU cache misuse for BlobFileCache causes significant performance degradation

[wangnengjie]

Defaultly, LRU cache uses a policy kFullChargeCacheMetadata which will take memory usage of meta into account. While using it as file cache where we hope 1 file takes only 1 charge, we should use the policy kDontChargeCacheMetadata. Here is where the cache instance of BlobFileCahe is created: https://github.com/tikv/titan/blob/ff85fd19e6268c57b3b01c45aeae011299c1d542/src/blob_file_set.cc#L22-L26

Under such setting, each file will be count as 89 charge. In my test case, i set max_open_files to 950 so only 10 files can be cached in LRU. Thus most of Get operations need to open a blob file again and again and i observed significant performance degradation.

file_number: 54 cacheusage: 89 total insert count: 1
file_number: 53 cacheusage: 178 total insert count: 2
file_number: 197 cacheusage: 267 total insert count: 3
file_number: 109 cacheusage: 356 total insert count: 4
file_number: 149 cacheusage: 445 total insert count: 5
file_number: 6 cacheusage: 534 total insert count: 6
file_number: 85 cacheusage: 623 total insert count: 7
file_number: 5 cacheusage: 712 total insert count: 8
file_number: 73 cacheusage: 801 total insert count: 9
file_number: 30 cacheusage: 890 total insert count: 10
file_number: 88 cacheusage: 890 total insert count: 11
file_number: 118 cacheusage: 890 total insert count: 12 

We should use it like this:

auto file_cache_size = db_options_.max_open_files;
if (file_cache_size < 0) {
  file_cache_size = kMaxFileCacheSize;
}
LRUCacheOptions co;
co.capacity = file_cache_size;
co.metadata_charge_policy = kDontChargeCacheMetadata;
file_cache_ = NewLRUCache(co);

[Connor1996]

I spotted the blob file cache hit rate was low before and didn't dig out the reason. That explains the reason. Thanks for your feedback!

[Connor1996]

Would you like to fix it? If not, I'll file a PR to fix it.

[wangnengjie]

@Connor1996 hello, i sent a pr to fix this. You can take a moment to look at it. Just a few lines