search

xichen2020 / eventdb

Storage for discrete events such as log events or tracing events
Apache License 2.0
9 stars 1 forks source link

OOM killer shoots eventdb after ~ 2 hours #128

Open notbdu opened 5 years ago

notbdu commented 5 years ago

When running eventdb on logging-spare-1t01-sjc1, it appears that the OOM killer shoots the process after approximately 2 hours of runtime.

Should do some memory profiling to see what the issue is.

notbdu commented 5 years ago

After some memory profiling, these are the offending areas w/ seemingly unbounded memory growth over the span of 3 separate heap dumps:

parser/json/(*parser) parseString: 5 GB -> 9.7 GB -> 21.6 GB -> parser/json/(*parser) parseStringAsRaw: 4.4 GB -> 9 GB -> 20.6 GB

server/http/handlers/(Service) Write: 12.8 GB -> 25.7 GB -> 115.9 GB -> ioutil ReadAll: 12.8 GB -> 25.7 GB -> 115.9 GB

notbdu commented 5 years ago

Heap 1: heap1

Heap 2: heap2

Heap 3: heap3

notbdu commented 5 years ago

Up until the point that the eventdb was shot, ~150 GB of data was written to it. That mirrors the memory held by the server/http/handlers/(Service) Write call. My suspicion is that the memory is not being GC'ed at all due to some persistently held references to the original byte slice.

Each data slice in write gets split into individual documents. But each document holds a reference to a portion of the original byte slice.

cw9 commented 5 years ago

Can you upload the profile here?