search

pinterest / secor

Secor is a service implementing Kafka log persistence
Apache License 2.0
1.84k stars 543 forks source link

Kuberntes OOM Error. Very High memory usage > 15gb #2129

Open josekidengan opened 3 years ago

josekidengan commented 3 years ago

Kubernetes memory usage is spiking more than 15Gb, but local run on ubuntu takes much less. I figured out that Kubernetes does not use swaps. However, I could not find any specific parameter to control this spiking of memory in case a swap is not present. Even At 2 threads, memory usage spikes to ~15GB.

HenryCaiHaiying commented 3 years ago

You can use 'jmap -histo' to print out the number of objects per class, run it a few times to see which class of objects keep increasing.

On Tue, Jun 29, 2021 at 9:21 AM Jose Thomas @.***> wrote:

Although secor is working, it looks like the memory usage is constantly increasing over time eventually shuts off the system. All I could sense is an info (org.apache.hadoop.io.compress.CodecPool) INFO Got brand-new compressor [.gz]

It looks like the object is being created without recycling and memory leaks is causing the system to shut down. Has anyone got a similar issue

Related config secor.compression.codec=org.apache.hadoop.io.compress.GzipCodec

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/2129, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP77RSLDRSRNRNRTKSNLTVHXILANCNFSM47QQAVHA .

josekidengan commented 3 years ago

You can use 'jmap -histo' to print out the number of objects per class, run it a few times to see which class of objects keep increasing. … On Tue, Jun 29, 2021 at 9:21 AM Jose Thomas @.***> wrote: Although secor is working, it looks like the memory usage is constantly increasing over time eventually shuts off the system. All I could sense is an info (org.apache.hadoop.io.compress.CodecPool) INFO Got brand-new compressor [.gz] It looks like the object is being created without recycling and memory leaks is causing the system to shut down. Has anyone got a similar issue Related config secor.compression.codec=org.apache.hadoop.io.compress.GzipCodec — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2129>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP77RSLDRSRNRNRTKSNLTVHXILANCNFSM47QQAVHA .

I update the comments and description with my insights.