Memory usage growing continuously

[cgarciaarano]

Hi,

we've deployed the last version (commit 9f4b04fe9ad1b6a0b158c55fbc2bdda39161d301) and after some days working, our monitoring system shows a continuous growth of memory usage by the captagent process.

We're running Ubuntu 12.04, compiled it with gcc, as this guide explains.

Here's the trend of memory usage (in green), showing a sustained growth...(The first valleys are restarts)

We will mantain the process alive for some days, to ensure that this behavior persists.

Regards

[adubovikov]

Hi, thank you for your report. Can you please also share the captagent's config ?

Wbr, Alexandr

[cgarciaarano]

Hi,

it's pretty much a default config, we've just changed a few things.

<?xml version="1.0"?>

<document type="captagent/xml">

        <configuration name="core.conf" description="CORE Settings">
          <settings>
            <param name="debug" value="3"/>
            <param name="daemon" value="true"/>
            <param name="syslog" value="true"/>
            <param name="pid_file" value="/var/run/captagent.pid"/>
            <param name="path" value="/usr/local/lib/captagent/modules"/>
          </settings>
        </configuration>

        <configuration name="modules.conf" description="Modules">
          <modules>
                <load module="core_hep"/>
                <load module="proto_uni"/>
                <load module="proto_rtcp"/>
                <load module="capt_cli"/>
          </modules>
        </configuration>

        <!-- CORE MODULES -->

        <configuration name="core_hep.conf" description="HEP Socket">
          <settings>
            <param name="version" value="3"/>
            <param name="capture-host" value="HOSTNAME"/>
            <param name="capture-port" value="9060"/>
            <param name="capture-proto" value="udp"/>
            <param name="capture-id" value="4004"/>
            <param name="capture-password" value="myHep"/>
            <param name="payload-compression" value="false" />
          </settings>
        </configuration>

        <!-- PROTOCOLS -->

        <configuration name="proto_uni.conf" description="UNI Proto Basic capture">
          <settings>
            <param name="port" value="5060"/>
            <!-- <param name="portrange" value="5060-5090"/> -->
            <!--
                use -D flag for pcap import
                use "any" for all interfaces in your system
            -->
            <param name="dev"  value="any"/>
            <param name="promisc" value="true"/>
            <!--
                comment it if you want to see all IPProto (tcp/udp)
            -->
            <!-- enable this if you want to get RTCP stats -->
            <param name="sip-parse"  value="true"/>
            <param name="rtcp-tracking"  value="true"/>
            <param name="ip-proto" value="udp"/>
            <param name="proto-type"  value="sip"/>
            <param name="filter" value="and not host HOSTNAME"/>
            <!--
                <param name="expire-timer" value ="120"/>
                <param name="expire-rtcp" value ="120"/>
            -->
          </settings>
        </configuration>

        <configuration name="proto_rtcp.conf" description="RTCP capture">
            <settings>
                <!-- <param name="portrange" value="5060-5090"/> -->
                <param name="dev"  value="any"/>
                <param name="promisc" value="true"/>
                <param name="debug" value ="false"/>
                <!-- <param name="filter" value="and not host HOSTNAME"/> -->
                <!-- <param name="rtcp-json" value="false"/> -->
                <!-- <param name="send-sdes" value="false"/> -->
                <!-- <param name="filter" value="and not src port 5099"/> -->
                <!-- <param name="vlan" value="false"/> -->
            </settings>
          </configuration>

        <!-- CLI  -->

        <configuration name="capt_cli.conf" description="CLI socket">
          <settings>
            <param name="cli-host" value="localhost"/>
            <param name="cli-port" value="8909"/>
            <param name="cli-password" value="12345"/>
          </settings>
        </configuration>

I have to say that this is our first captagent installation, so maybe it's not a version related issue.

Also, we're monitoring the packets per seconds connecting to the CLI each minute, via netcat. Maybe this could lead to any kind of memory leak.

If you need something more, let me know.

Thanks

[adubovikov]

Can you please temporally disable rtcp tracking and check it again ?

[cgarciaarano]

Done, i'll keep you posted

Thanks

2015-05-08 7:21 GMT+02:00 Alexandr Dubovikov notifications@github.com:

Can you please temporally disable rtcp tracking and check it again ?

— Reply to this email directly or view it on GitHub https://github.com/sipcapture/captagent/issues/9#issuecomment-100103260.

Carlos García Director de Operaciones Tel. 695 624 167 - 902 620 100 www.stoneworksolutions.net

AVISO DE CONFIDENCIALIDAD Tanto este mensaje como todos los posibles documentos adjuntos al mismo son confidenciales y están dirigidos exclusivamente a los destinatarios de los mismos. Por favor, si Ud no es uno de dichos destinatarios, notifíquenos este hecho y elimine el mensaje de su sistema. Queda prohibida la copia, difusión o revelación de su contenido a terceros sin el previo consentimiento por escrito del remitente. En caso contrario, vulnerarán la legislación vigente

[adubovikov]

Hi,

are you talking about SIP or RTCP packets ?

On 2015-05-08 10:10, cgarciaarano wrote:

Hi,

after commenting out rtcp configuration we're not getting packages from that probe in our homer server. Maybe I messed up configuration:

     <!-- CLI  -->

— Reply to this email directly or view it on GitHub https://github.com/sipcapture/captagent/issues/9#issuecomment-100145077.

[cgarciaarano]

Hi Alexandr,

after one day running without the RTPC module, the behaviour looks the same, growing continuously. Even it shows the same pace, taking 1,8% of memory per day. The host has 1 GB RAM, so it's not really a problem, but we don't like to setup a cronjob to restart the process periodically.

Regards

[adubovikov]

ok, now please disable SIP-parsing (set to false).

thank you!

[cgarciaarano]

Hi Alexendr,

it's still growing, but at slower pace. Find attached a period of 1 day and 19 hours, when it just grew 0,3%. (blue line is absent, as there are no packets being processed)

The only modules loaded are:

• CLI
• HEP Core

Thanks

[cgarciaarano]

More on this issue.

We've seen that the memory consumption is growing at the same pace in processes that sniff SIP traffic with different usage patterns.

So it looks like the "alleged" memory leak has nothing to do with sniffing. ¿There are any periodic task running? ¿Reconnections or something similar?

Regards

[adubovikov]

there should not be any additional threads if you set sip-parse and rtcp-tracking to false.

[AlessioCasco]

Hello,

I'm experiencing the same issue with latest revision of captagent 6 (cloned 5 days ago), where there is a continuous growth of memory usage as reported by @cgarciaarano.

I tried disabling all the modules I can without any luck:

<?xml version="1.0"?>
<document type="captagent/xml">
    <configuration name="core.conf" description="CORE Settings" serial="2014024212">
        <settings>
        <param name="debug" value="3"/>
        <param name="version" value="2"/>
        <param name="serial" value="2014056501"/>
        <param name="uuid" value="00781a4a-5b69-11e4-9522-bb79a8fcf0f3"/>
        <param name="daemon" value="true"/>
        <param name="syslog" value="true"/>
        <param name="pid_file" value="/var/run/captagent.pid"/>
        <param name="module_path" value="/usr/lib64/captagent/modules"/>
        <param name="config_path" value="/etc/captagent"/>
        <param name="capture_plans_path" value="/etc/captagent/captureplans"/>
        <param name="backup" value="/etc/captagent/backup"/>
        <param name="chroot" value="/var/lib/captagent"/>
        </settings>
    </configuration>
    <configuration name="modules.conf" description="Modules">
        <modules>
        <load module="transport_hep" register="local"/>
        <load module="database_hash" register="local"/>
        <load module="protocol_sip" register="local"/>
        <load module="socket_pcap" register="local"/>
        <!--
        ***We don't use these modules***
        <load module="socket_raw" register="local"/>
        <load module="transport_json" register="local"/>
        <load module="protocol_rtcp" register="local"/>
        <load module="interface_http" register="local"/>
        <load module="database_redis" register="local"/>
        <load module="socket_pfring" register="local"/>
        -->
        </modules>
    </configuration>
</document>

I'm using Debian GNU/Linux 7 and 8

Looks like that more traffic captagent is parsing, faster the memory is growing

Thanks Alessio

[apogrebennyk]

I noticed the same issue as Alessio, will try disabling the modules at next occasion. Right now I have reverted to 4.2.0.

[lmangani]

We are investigating, thanks for the reports guys, updates will follow

[AlessioCasco]

Hello Guys! Any news on this?

Thanks Alessio

[adubovikov]

Hello Alessio,

yes, we will release the new version on this weekend.

Wbr, Alexandr

On 17 February 2016 at 12:46, Alessio Casco notifications@github.com wrote:

Hello Guys! Any news on this?

Thanks Alessio

— Reply to this email directly or view it on GitHub https://github.com/sipcapture/captagent/issues/9#issuecomment-185168470.

[AlessioCasco]

Hello Guys! Just a confirmation, has the new version been released?

Thanks

[AlessioCasco]

Hello Guys!

FYI I'm testing version 6.1.0, will let you know the results in a couple of days

Alessio

[adubovikov]

Thank you Alessio

[AlessioCasco]

Hello Guys!

Good news, after almost two weeks, version 6.1.0 is behaving great!

PID     USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM  TIME+    COMMAND
10249   root      20   0  115.1m  14.6m  14.3m S   3.6  0.2  39:54.06 captagent

I'm using a very simple conf (same as the previous one) and so far so good.

Let me know if you need more stuff/info from me

Thanks again for the fix Alessio

[lmangani]

Thanks @AlessioCasco for testing and taking time to update us back!

[adubovikov]

Thank you Alessio, can you please also activate protocol_rtcp and check how is going ?

I will move 6.1 to master repository.

[AlessioCasco]

Actually has been already activated since day one:

• protocol_rtcp.xml

<?xml version="1.0"?>
<document type="captagent_module/xml">
    <module name="protocol_rtcp" description="RTCP Protocol" serial="blablabla">
    <profile name="proto_rtcp" description="PROTO RTCP" enable="true" serial="blablabla">
        <settings>
        <param name="rtcp-version" value="2"/>
        </settings>
    </profile>
    </module>
</document>

• rtcp_capture_plan.cfg

capture[pcap] {

    # here we can check source/destination IP/port, message size
    if(msg_check("size", "100")) {

        if(is_rtcp()) {

            if(is_rtcp_exist()) {

                #Convert to JSON if needed.
                if(parse_rtcp_to_json()) {

                    #Can be defined many profiles in transport_hep.xml
                    if(!send_hep("hepsocket")) {
                        clog("ERROR", "Error sending HEP!!!!");
                    }

                } else {
                    clog("ERROR", "couldn't parse RTCP to json");
                }

            } else {
                clog("ERROR", "Couldnot find this call");
            }
        } else {
            clog("ERROR", "This is not RTCP");
        }
        }
    drop;
}

[adubovikov]

this is very good, but the problem is that you didn't load protocol_rtcp in the captagent.xml

second, please change if(msg_check("size", "100")) to if(msg_check("size", "30"))

[AlessioCasco]

The module is loaded, will change the size to 30 and let you know

    <configuration name="modules.conf" description="Modules">
        <modules>
            <load module="transport_hep" register="local"/>
            <load module="protocol_sip" register="local"/>
            <load module="protocol_rtcp" register="local"/>
            <load module="socket_rtcpxr" register="local"/>
            <load module="socket_pcap" register="local"/>
            <load module="socket_raw" register="local"/>
            <!--
            ***We don't use these modules***
            <load module="database_hash" register="local"/>
            <load module="transport_json" register="local"/>
            <load module="protocol_rtcp" register="local"/>
            <load module="interface_http" register="local"/>
            <load module="database_redis" register="local"/>
            <load module="socket_pfring" register="local"/>
            -->
        </modules>
    </configuration>

[adubovikov]

Ok, thank you! My comment was to your previous loaded config file without protocol_rtcp module.

[AlessioCasco]

Hello guys,

I believe we can close this issue, captagent has been running since last comment and the memory still looks ok.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 
11111 root      20   0  115.1m  14.1m  13.8m S   0.0  0.2  78:21.28 captagent

Can we consider closing this as dangerous demo from the #KamailioWorld :-)

[adubovikov]

Thank you Alessio!