search

grobian / carbon-c-relay

Enhanced C implementation of Carbon relay, aggregator and rewriter
Apache License 2.0
380 stars 107 forks source link

Memory consumption issue after upgrade to Centos 8 #434

Closed Rohlik closed 2 years ago

Rohlik commented 3 years ago

Hello there 👋🏼, after what we hit on #432 we decided to upgrade our production server to Centos 8 and enable native SSL in Carbon-C-relay, but since then we have problem with memory consumption. Number of received metrics should be same as before upgrade to CO8 but memory usage is much more higher and it increase over time until carbon-c-relay gets kill by OOM.

On that virtual server only carbon-c-relay is running. This behavior is happening on all of our Centos 8 servers with carbon-c-relay native SSL enabled.

Histogram from one of our server shows state before upgrade and after (7.6. afternoom): image

Process:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                                                               
 161082 carbon-+  20   0  236.2g  30.3g   5252 S 155.4  77.2   3413:06 carbon-c-relay

CMD: /usr/bin/carbon-c-relay -f /etc/carbon-c-relay.conf -c -_:#^ -b 2500 -q 7500000 -l /mnt/log/carbon-c-relay/carbon-c-relay.log

Kernel (uname -r): 4.18.0-240.22.1.el8_3.x86_64 OS: CentOS Linux release 8.3 Carbon-c-relay: carbon-c-relay-3.7.2

grobian commented 3 years ago

feels like it leaks a handle or something, thanks

grobian commented 3 years ago

Do you frequently reload the configuration by chance? That's the only leak I've been able to find sofar.

Rohlik commented 3 years ago

@grobian Well, we are using Puppet to delivery changes every hour, but we don't reload configuration if there is no update 🤔 . Anyways, we had same Puppet even on Centos 7 and there wasn't any issues with memory leaks as you can see on graph.

grobian commented 2 years ago

This looks a lot like issue #439, but there CentOS 7 is used, and also there I seem to be completely unable to reproduce the problem :(

berrfred commented 2 years ago

I suggest you give a try to the latest release 3.7.3, that solved our SSL memory leak issue on CentOS 7.

Rohlik commented 2 years ago

Thx, I will do that 🤞🏽 .

Rohlik commented 2 years ago

Well i wasn't successful as @grobian recently bump version of autoconf to 2.71 and latest version available for Centos 8 is 2.69 😞 , so I cannot build RPM for our environment.

grobian commented 2 years ago

hmmmm...

Did you use the tar that I uploaded to the release? It shouldn't require autotools. If you want to use the github zip, you'll have to use ./configure --disable-maintainer-mode and probably touch some files that would be generated by bison and flex.

Rohlik commented 2 years ago

Thanks @grobian , you are right. It's not needed for tar archive. Thank you. We will proceed with update carbon-c-relay.

Rohlik commented 2 years ago

Problem looks solved after upgrade to latest version 2.7.3 in our DEV env. image