databendlabs / databend

๐——๐—ฎ๐˜๐—ฎ, ๐—”๐—ป๐—ฎ๐—น๐˜†๐˜๐—ถ๐—ฐ๐˜€ & ๐—”๐—œ. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
https://docs.databend.com
Other
7.71k stars 732 forks source link

feat: introduce columnar segment info cache #16505

Open SkyFan2002 opened 2 days ago

SkyFan2002 commented 2 days ago

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

The test scenario

Cache SegmentInfo:

scenario 3000 SegmentInfo, 1000 block per seg , pid 2688957, base memory 224849920

            cache item type : SegmentInfoCache,
            number of cached items 3000,
            mem usage(B):+26773348352,
            mem usage(GB)+24.93462371826172

Cache ColumnarSegmentInfo:

scenario 3000 SegmentInfo, 1000 block per seg , pid 2688450, base memory 225886208

            cache item type : ColumnarSegmentInfoCache,
            number of cached items 3000,
            mem usage(B):+8549457920,
            mem usage(GB)+7.962303161621094

Tests

Type of change


This change isโ€‚Reviewable