splunk / splunk-library-javalogging

Splunk logging appenders for popular Java Logging frameworks
http://dev.splunk.com/view/splunk-logging-java/SP-CAAAE2K
Apache License 2.0
135 stars 175 forks source link

Memory issue when SPLUNK is not available #249

Closed MulesoftDevelop closed 2 years ago

MulesoftDevelop commented 2 years ago

Hi Team,

we are using the SPLUNK appender in Mulesoft with these settings:

    <SplunkHttp
        name="splunk"
        url="${sys:SPLUNK_URL}"
        token="${sys:SPLUNK_TOKEN}"
        host="${project.artifactId}"
        index="myindexname_${sys:mule_env}"
        source="${project.artifactId}"
        sourcetype="mule-application"
        messageFormat="text"
        batch_interval="0"
        batch_size_bytes="0"
        batch_size_count="0"
        disableCertificateValidation="false">
        <PatternLayout pattern="%m%n" />
    </SplunkHttp>

Unfortunately we had some downtimes of SPLUNK. During these downtimes the memory consumption increased until the applications crashed. We have also tried different settings for batch_size_count and batch_interval but this didn't help.

Is there any way of configuring the appender to drop the logs if it is not possible to deliver then to SPLUNK? The goal is to keep the application stable even if logging is not working.

Best Regards, Sebastian Liepe

MulesoftDevelop commented 2 years ago

Hi Team,

we have tried to set different timeout parameters. This has improved the situation but the issues is not solved. The screenshot the behavior as well as the used settings:

grafik

During the test we processed about 2k messages which are producing 4k log entries per minute.

BR, Sebastian

bparmar-splunk commented 2 years ago

Hi @MulesoftDevelop,

Thank you for raising this issue.

For Appender settings, please check this link for reference. Property of batch_size_count with recommended value is mentioned on above link (for production).

Also, in terms of changing timeout settings, we would recommend to update logging library version to 1.11.5 which includes improvements related to timeout and error call backs.

Please check with this updated version and let us know if you still face any issue.

Thank you.

bparmar-splunk commented 2 years ago

Hi @MulesoftDevelop, Based on your confirmation to internal team, issue has been resolved.

Hence, closing this issue.