danielealbano / cachegrand

cachegrand - a modern data ingestion, processing and serving platform built for today's hardware
BSD 3-Clause "New" or "Revised" License
975 stars 34 forks source link

Feat avoid secondary lzf buffer #392

Closed danielealbano closed 1 year ago

danielealbano commented 1 year ago

This PR changes the lzf compression support when dumping the RDB snapshot to avoid using a secondary buffer on the stack.

It also update the code (although it's kept disabled) that compresses data with LZF to use the new storage buffered channels.

The library in use requires way too much stack space (1MB) to compress 256 bytes of data, it's an absolute waste of memory. Once we will switch to a different library this functionality will be re-enabled.