Memory Manager: Report to the log per cf / level / block type stats

[udi-speedb]

• Report the following per block cache:
  • Per CF sharing the block cache in the DB, report the following information for the data blocks, index blocks, filter blocks, and compression dictionary blocks:
  • Separately for Level-0, Middle Level (0 < level < Last-Level-With-Data), and Last-Level-With-Data Level:
    • CURRENT Total size in cache
    • CURRENT Total pinned size in cache
    • Number of hits in the last dump interval
    • Total Number of hits
    • Number of misses in the last dump interval
    • Total number of misses

The information will be printed to the log according to the stats_dump_period_sec option.

The feature currently only supports the LRU-Cache

[udi-speedb]

@hilikspdb - It seems there is no dedicated cache entry role for compression dictionaries. There is:

  // Compression dictionary building buffer's charge to account for
  // its memory usage
  kCompressionDictionaryBuildingBuffer,

that may be used to charge to cache the memory used to build the block, but that is not the role used when inserting the compression dictionary block into the cache. According to my checks, the role used is CacheEntryRole::kOtherBlock

[hilikspdb]

you can add this ... currently the "metadata block" (which contains the delete range tombstone) and the dictionary are accounted as "others"

On Wed, Oct 11, 2023 at 7:38 PM udi-speedb @.***> wrote:

@hilikspdb https://github.com/hilikspdb - It seems there is no dedicated cache entry role for compression dictionaries. There is:

// Compression dictionary building buffer's charge to account for // its memory usage kCompressionDictionaryBuildingBuffer,

that may be used to charge to cache the memory used to build the block, but that is not the role used when inserting the compression dictionary block into the cache. According to my checks, the role used is CacheEntryRole::kOtherBlock

— Reply to this email directly, view it on GitHub https://github.com/speedb-io/speedb/issues/660#issuecomment-1758083116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZKTXGEWJS5FN5XJM3DW573X63DR3ANCNFSM6AAAAAA4JF4KZQ . You are receiving this because you were mentioned.Message ID: @.***>

[udi-speedb]

you can add this ... currently the "metadata block" (which contains the delete range tombstone) and the dictionary are accounted as "others" … On Wed, Oct 11, 2023 at 7:38 PM udi-speedb @.> wrote: @hilikspdb https://github.com/hilikspdb - It seems there is no dedicated cache entry role for compression dictionaries. There is: // Compression dictionary building buffer's charge to account for // its memory usage kCompressionDictionaryBuildingBuffer, that may be used to charge to cache the memory used to build the block, but that is not the role used when inserting the compression dictionary block into the cache. According to my checks, the role used is CacheEntryRole::kOtherBlock — Reply to this email directly, view it on GitHub <#660 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZKTXGEWJS5FN5XJM3DW573X63DR3ANCNFSM6AAAAAA4JF4KZQ . You are receiving this because you were mentioned.Message ID: @.>

@hilikspdb - Do you mean that I should maintain counters for kOtherBlock?

[hilikspdb]

It is counted toward the cache as part of the static pinning under the others. It is always pinned even if there is not enough space..

On Wed, 11 Oct 2023, 20:15 udi-speedb, @.***> wrote:

you can add this ... currently the "metadata block" (which contains the delete range tombstone) and the dictionary are accounted as "others" … <#m3023917537616854293> On Wed, Oct 11, 2023 at 7:38 PM udi-speedb @.> wrote: @hilikspdb https://github.com/hilikspdb https://github.com/hilikspdb https://github.com/hilikspdb - It seems there is no dedicated cache entry role for compression dictionaries. There is: // Compression dictionary building buffer's charge to account for // its memory usage kCompressionDictionaryBuildingBuffer, that may be used to charge to cache the memory used to build the block, but that is not the role used when inserting the compression dictionary block into the cache. According to my checks, the role used is CacheEntryRole::kOtherBlock — Reply to this email directly, view it on GitHub <#660 (comment) https://github.com/speedb-io/speedb/issues/660#issuecomment-1758083116>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZKTXGEWJS5FN5XJM3DW573X63DR3ANCNFSM6AAAAAA4JF4KZQ https://github.com/notifications/unsubscribe-auth/AZKTXGEWJS5FN5XJM3DW573X63DR3ANCNFSM6AAAAAA4JF4KZQ . You are receiving this because you were mentioned.Message ID: @.>

@hilikspdb https://github.com/hilikspdb - Do you mean that I should maintain counters for kOtherBlock?

— Reply to this email directly, view it on GitHub https://github.com/speedb-io/speedb/issues/660#issuecomment-1758141503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZKTXGHYWECSUN7ASPIX7JLX63H4ZANCNFSM6AAAAAA4JF4KZQ . You are receiving this because you were mentioned.Message ID: @.***>

[udi-speedb]

To sum this up: Compression Dictionary blocks will remain under the kOtherBlock cache entry role and will be reported under that role.