CH-anhngo commented 5 years ago

syslog-ng

Looks like we got the same issue with #2380. Even we tweak the log_fifo_size to really low compare with our memory in the box the memory usage keeps climbing until it got OOM-killer and thing start from 0 to OOM-killer again and again.

(Please, remove the unnecessary notes, including this one.)

Version of syslog-ng

v 3.17.2

output of syslog-ng --version
syslog-ng 3 (3.17.2)
Config version: 3.17
Installer-Version: 3.17.2
Revision: 
Module-Directory: /opt/lib/syslog-ng
Module-Path: /opt/lib/syslog-ng
Available-Modules: tfgetent,tags-parser,linux-kmsg-format,csvparser,syslogformat,hook-commands,graphite,add-contextual-data,disk-buffer,date,cryptofuncs,json-plugin,affile,xml,pseudofile,snmptrapd-parser,sdjournal,map-value-pairs,system-source,appmodel,basicfuncs,cef,afuser,confgen,dbparser,stardate,afsocket,kvformat,afprog
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Systemd: on

Platform

Coreos (Name and version of OS)

Debug bundle

Create a debug bundle on your system with the syslog-ng-debun script which is included in the syslog-ng package.

Overwiew of the CLI options of syslog-ng-debun: -r: run actual information gathering -d: run syslog-ng in debug mode -p: perform packet capture -s: do strace -t: timeout period for running debug/pcap/strace -w: wait period before starting debug mode -l: light information gathering (respects privacy) -R: alternate installation directory for syslog-ng

$ syslog-ng-debun -r

Syslog-NG DEBUg buNdle generator
Start environment detection
Linux-type FHS detected
Unknown or unsupported Linux distribution!

Operating System Name: Linux
Unknown Distro, perhaps unsupported
Unknown Distro, perhaps unsupported
libsemanage.semanage_create_store: Could not access module store at /var/lib/selinux/mcs, or it is not a directory. (Read-only file system).
libsemanage.semanage_direct_connect: could not establish direct connection (Read-only file system).
semodule:  Could not connect to policy handler

Start general info collection
System's full uname: Linux logserver001 4.19.43-coreos #1 SMP Wed May 15 07:18:39 -00 2019 x86_64 Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz GenuineIntel GNU/Linux
Getting network-interface information: Success
Getting network routes: Success
Getting DNS resolution-related information: Done
List all processes
Mount and disk free info collection

Start Syslog-specific info collection
Copy configs from /etc/syslog-ng
/opt/sbin/syslog-ng-debun: line 654: cpio: command not found
Gathering PKI information... done.
cat: /var/lib/syslog-ng/syslog-ng.pid: No such file or directory
SVpid:  SNGpid:  Chpids: 
Syslog-ng's exact version: /opt/sbin/syslog-ng-debun: line 712: /usr/sbin/syslog-ng: No such file or directory
/usr/sbin/syslog-ng-ctl stats
/usr/sbin/syslog-ng-ctl query get *
/usr/sbin/syslog-ng-ctl show-license-info
/usr/sbin/syslog-ng-ctl credentials status
/opt/sbin/syslog-ng-debun: line 729: /usr/sbin/syslog-ng: No such file or directory
/opt/sbin/syslog-ng-debun: line 747: cpio: command not found
ldd: /usr/sbin/syslog-ng: No such file or directory
/opt/sbin/syslog-ng-debun: line 790: /tmp/syslog.debun.logserver1.-05-24_22-33.31137/syslog.ldpkg: No such file or directory
Detecting init system: systemd detected...
Generating second batch of statistics
/usr/sbin/syslog-ng-ctl stats
/usr/sbin/syslog-ng-ctl query get *

Generating hashes... done.
Debug Bundle generation: Done.
Terminating live message watcher: /opt/sbin/syslog-ng-debun: line 273: 31145 Terminated              tail -f ${tmpdir}/syslog-ng.debun.txt 1>&3  (wd: ~)

Your debug bundle will be stored at /tmp/syslog.debun.logserver001-05-24_22-33.31137.tgz

Issue

Failure

(Backtrace, error messages or detailed description of failure comes here.) (To create debug bundles, use syslog-ng-debun. Details of its usage can be found on the documentation page)

$ gdb syslog-ng
 run

Steps to reproduce

start syslog-ng
start fluentd to read from syslog-ng buffer to upload to s3

Configuration

$ cat /path/to/syslog-ng.conf
@version: 3.17
@include "/opt/etc/scl.conf"

options {
    flush_lines(0);
    chain_hostnames(off);
    keep_hostname(yes);
    log_fifo_size(100000);
    stats_freq(43200);
    time_reopen(10);
    ts_format(iso);
    use_dns(no);
    use_fqdn(no);
};

destination d_fluent_s3 {
    network("127.0.0.1" port(8898)
        disk-buffer(
                    mem-buf-size(163840000)
                    disk-buf-size(10GiB)
                    reliable(yes)
                    dir("/var/lib/data/buffer")
                   )
    );
};

source s_network {
    network(
        max-connections(1000)
        port(7858)
        log_msg_size(65536)
    );
};

log {
    source(s_network);
    destination(d_fluent_s3);
};

(or gist URL)

Input and output logs (if possible)

gaborznagy commented 5 years ago

Hi @CH-anhngo,

Please not that #2380 was not a bug, rather a question about syslog-ng's memory usage. It turned out to be simply a destination was using log-fifo - a memory buffer - and there were messages in the queue.

In your case you are using reliable disk-buffer, which stores every incoming message on the disk, thus memory buffering is not done in memory.

Can you please give us some output of the amount of memory used when you see syslog-ng is leaking memory?