search

kelindar / column

High-performance, columnar, in-memory store with bitmap indexing in Go
MIT License
1.44k stars 57 forks source link

possible memory leak when taking a snapshot #85

Open ami-m opened 1 year ago

ami-m commented 1 year ago

I created a poc with column, it got hit with incoming data for a bout an hour, and after that I ran a loop that created a snapshot of the data once a minute.

I monitored the memory consumption and the memory usage jumped (and was not released) every time a snapshot was taken.

Other than closing the underlying writer, is there anything else I should have released?

if not, then I suspect a memory leak. pprof points the finger at the column.grow and s2.NewWriter as possible culprits.

kelindar commented 1 year ago

Will need to investigate this, do you have a test case?

tanzegen commented 10 months ago

Yes, I also have this problem, construct a loop to execute the snapshot, you will find that there is a fixed increase in memory, the Snapshot method creates an s2.NewWriter, does not seem to free it