search

syslog-ng / syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
https://www.syslog-ng.com
Other
2.12k stars 471 forks source link

Syslog-ng stopped pushing logs to Elasticsearch #2349

Closed JawadKM closed 5 years ago

JawadKM commented 6 years ago

Hi, I have my syslog-ng pushing data to Elasticsearch, it was working perfectly fine but then suddenly stopped working. Below is the example of the config in syslog-ng "source and destination of my log file"

Client server; source s_apache{ file("/opt/int/apache/logs/cat.log"); };

destination d_syslog_tcp { ewmm(ip("10.10.0.3") transport("tcp") port(514) ); }; log { source(s_apache); destination(d_syslog_tcp); };

Log server @version:3.17 @include "scl.conf" options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_network { ewmm(transport(tcp) port(514)); }; destination d_elastic { elasticsearch2 ( client-lib-dir(/usr/share/elasticsearch/lib) cluster("syslog-ng") client-mode("http") index("syslog-ng") type("test") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ) }; log { source(s_network); destination(d_elastic); flags(flow-control); }; Elasticsearch

curl 'localhost:9200/_cat/indices?bytes=b' | sort -rnk8 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 164 100 164 0 0 2118 0 --:--:-- --:--:-- --:--:-- 2129 yellow open syslog-ng nKG09rgtTiumAW5vZPbySw 5 1 65378579 0 9783890061 9783890061 green open .kibana aibdk5WFTT65EiOh8ed0gg 1 0 3 0 20306 20306

Note: yellow open syslog-ng nKG09rgtTiumAW5vZPbySw 5 1 65378579 0 9783890061 9783890061 Blod number is exacitally the same number shown in Kibaba number of documents.

image

Any help in this regards will be much appreciated.

szemere commented 6 years ago

Hello, since your setup worked before, as a first step I would like to identify where the messages get stuck currently. Can you use syslog-ng-ctl stats to see which counter is changing and how much? (especially interested in: processed, dropped, queued)

Br, Laci

faxm0dem commented 6 years ago

There is an open bug #1993 that causes an infinite loop when ES doesn't parse a message. Can you try flush-limit(1) to confirm this is the same issue?

JawadKM commented 6 years ago

@szemere following are the results of syslog-ng-ctl stats

Client SourceName;SourceId;SourceInstance;State;Type;Number center;;received;a;processed;12902 source;s_e_A_M_log;;a;processed;361 source;s_apache;;a;processed;451 source;s__STSA;;a;processed;0 center;;queued;a;processed;12902 global;scratch_buffers_count;;a;queued;77309411328 source;s_authentication_audit;;a;processed;47 destination;d_syslog_tcp;;a;processed;12902 global;payload_reallocs;;a;processed;5284 global;sdata_updates;;a;processed;0 source;s_H_xyz;;a;processed;133 dst.network;d_syslog_tcp#0;tcp,10...:514;a;dropped;5361 dst.network;d_syslog_tcp#0;tcp,10...:514;a;processed;12902 dst.network;d_syslog_tcp#0;tcp,10...:514;a;queued;0 dst.network;d_syslog_tcp#0;tcp,10...:514;a;written;7541 source;s_sts_A_M;;a;processed;0 global;scratch_buffers_bytes;;a;queued;0 source;s_Fed;;a;processed;11910 global;msg_clones;;a;processed;0 source;s_E_A_M;;a;processed;0

Log Server SourceName;SourceId;SourceInstance;State;Type;Number source;s_network;;a;processed;11073 destination;d_elastic;;a;processed;7541 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;dropped;2541 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;processed;7541 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;queued;0 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;written;5000 center;;received;a;processed;11073 center;;queued;a;processed;7541 global;scratch_buffers_count;;a;queued;10505490006049 global;payload_reallocs;;a;processed;1643 global;sdata_updates;;a;processed;7541 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;0

And after some time this is the result of the same command

syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number source;s_network;;a;processed;137 destination;d_elastic;;a;processed;137 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;dropped;37 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;processed;137 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;queued;100 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;written;0 center;;received;a;processed;137 center;;queued;a;processed;137 global;scratch_buffers_count;;a;queued;158913789984 global;payload_reallocs;;a;processed;18 global;sdata_updates;;a;processed;137 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;0

JawadKM commented 6 years ago

@faxm0dem You want me to inclide "flush-limit(1)" in global options?

JawadKM commented 6 years ago

@faxm0dem I included "flush-limit (1)" as follows;

destination d_elastic { elasticsearch2 ( flush-limit ("1") client-lib-dir(/usr/share/elasticsearch/lib) cluster("syslog-ng") client-mode("http") index("syslog-ng") type("test") template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ) };

But the issue still remains, looking forward to your help.

szemere commented 6 years ago

@JawadKM thank you for the output. It shows me that the logs are fetched on the client, and successfully forwarded towards the server. But the server can not send it to elastic (queued: 100, processed: 137, dropped: 37) after the first 100 message the outgoing queue filled up, and syslog-ng has started to drop messages.

The next step will be to investigate the connection between Elasticsearch and syslog-ng(server). Can you start syslog-ng with the '-Fdev' options? Are there any error messages regarding the java destination in the output?

JawadKM commented 6 years ago

@szemere If you don't mind telling me the command to start syslog-ng with the "-Fdev" option, as I have been trying but falling.

JawadKM commented 6 years ago

@szemere These are the results I am getting

systemctl status syslg-ng (after stopping syslog-ng)

syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: inactive (dead) since Thu 2018-10-18 10:14:30 EDT; 6min ago Docs: man:syslog-ng(8) Process: 28969 ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid (code=exited, status=0/SUCCESS) Main PID: 28969 (code=exited, status=0/SUCCESS) Status: "Shutting down... (Thu Oct 18 10:14:30 2018"

Oct 18 10:14:11 openontario.ca syslog-ng[28969]: at org.syslog_ng.LogPipe.initProxy(LogPipe.java:64) Oct 18 10:14:11 openca syslog-ng[28969]: log4j:ERROR Ignoring configuration file [null]. Oct 18 10:14:12 open.ca syslog-ng[28969]: [main] INFO io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [http://localhost:9200] Oct 18 10:14:12 open.ca syslog-ng[28969]: [main] INFO io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager Oct 18 10:14:12 open.ca syslog-ng[28969]: [main] INFO io.searchbox.client.JestClientFactory - Using default GSON instance Oct 18 10:14:12 open.ca syslog-ng[28969]: [main] INFO io.searchbox.client.JestClientFactory - Node Discovery disabled... Oct 18 10:14:12 open.ca syslog-ng[28969]: [main] INFO io.searchbox.client.JestClientFactory - Idle connection reaping disabled... Oct 18 10:14:12 open.ca systemd[1]: Started System Logger Daemon. Oct 18 10:14:30 open.ca systemd[1]: Stopping System Logger Daemon... Oct 18 10:14:30 open.ca systemd[1]: Stopped System Logger Daemon.

syslog-ng -Fdev

syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' [2018-10-18T10:15:43.338474] Systemd is detected as the running init system; [2018-10-18T10:15:43.339328] Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1' [2018-10-18T10:15:43.340729] Adding include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2018-10-18T10:15:43.340741] Adding include file; filename='/usr/share/syslog-ng/include/scl/cim/adapter.conf', depth='2' [2018-10-18T10:15:43.340749] Adding include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2018-10-18T10:15:43.340755] Adding include file; filename='/usr/share/syslog-ng/include/scl/cisco/plugin.conf', depth='2' [2018-10-18T10:15:43.340762] Adding include file; filename='/usr/share/syslog-ng/include/scl/default-network-drivers/plugin.conf', depth='2' [2018-10-18T10:15:43.340768] Adding include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2018-10-18T10:15:43.340774] Adding include file; filename='/usr/share/syslog-ng/include/scl/ewmm/ewmm.conf', depth='2' [2018-10-18T10:15:43.340781] Adding include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2018-10-18T10:15:43.340787] Adding include file; filename='/usr/share/syslog-ng/include/scl/graylog2/plugin.conf', depth='2' [2018-10-18T10:15:43.340793] Adding include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2018-10-18T10:15:43.340799] Adding include file; filename='/usr/share/syslog-ng/include/scl/iptables/iptables.conf', depth='2' [2018-10-18T10:15:43.340806] Adding include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2018-10-18T10:15:43.340812] Adding include file; filename='/usr/share/syslog-ng/include/scl/loadbalancer/plugin.conf', depth='2' [2018-10-18T10:15:43.340818] Adding include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2018-10-18T10:15:43.340824] Adding include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2018-10-18T10:15:43.340830] Adding include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2018-10-18T10:15:43.340837] Adding include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2018-10-18T10:15:43.340844] Adding include file; filename='/usr/share/syslog-ng/include/scl/osquery/plugin.conf', depth='2' [2018-10-18T10:15:43.340851] Adding include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2018-10-18T10:15:43.340857] Adding include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2018-10-18T10:15:43.340864] Adding include file; filename='/usr/share/syslog-ng/include/scl/snmptrap/snmptrapd-source.conf', depth='2' [2018-10-18T10:15:43.340870] Adding include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2018-10-18T10:15:43.340877] Adding include file; filename='/usr/share/syslog-ng/include/scl/sudo/sudo.conf', depth='2' [2018-10-18T10:15:43.340883] Adding include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2018-10-18T10:15:43.340889] Adding include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2018-10-18T10:15:43.340896] Adding include file; filename='/usr/share/syslog-ng/include/scl/telegram/telegram.conf', depth='2' [2018-10-18T10:15:43.340902] Adding include file; filename='/usr/share/syslog-ng/include/scl/windowseventlog/plugin.conf', depth='2' [2018-10-18T10:15:43.340923] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2018-10-18T10:15:43.340983] Reading path for candidate modules; path='//usr/lib64/syslog-ng' [2018-10-18T10:15:43.341055] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libadd-contextual-data.so', module='add-contextual-data' [2018-10-18T10:15:43.341313] Registering candidate plugin; module='add-contextual-data', context='parser', name='add_contextual_data' [2018-10-18T10:15:43.341360] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libafamqp.so', module='afamqp' [2018-10-18T10:15:43.341617] Registering candidate plugin; module='afamqp', context='destination', name='amqp' [2018-10-18T10:15:43.341663] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libaffile.so', module='affile' [2018-10-18T10:15:43.341875] Registering candidate plugin; module='affile', context='source', name='file' [2018-10-18T10:15:43.341886] Registering candidate plugin; module='affile', context='source', name='pipe' [2018-10-18T10:15:43.341892] Registering candidate plugin; module='affile', context='source', name='wildcard_file' [2018-10-18T10:15:43.341899] Registering candidate plugin; module='affile', context='source', name='stdin' [2018-10-18T10:15:43.341906] Registering candidate plugin; module='affile', context='destination', name='file' [2018-10-18T10:15:43.341912] Registering candidate plugin; module='affile', context='destination', name='pipe' [2018-10-18T10:15:43.341940] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libafprog.so', module='afprog' [2018-10-18T10:15:43.342121] Registering candidate plugin; module='afprog', context='source', name='program' [2018-10-18T10:15:43.342131] Registering candidate plugin; module='afprog', context='destination', name='program' [2018-10-18T10:15:43.342156] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libafsocket.so', module='afsocket' [2018-10-18T10:15:43.342569] Registering candidate plugin; module='afsocket', context='source', name='unix-stream' [2018-10-18T10:15:43.342580] Registering candidate plugin; module='afsocket', context='destination', name='unix-stream' [2018-10-18T10:15:43.342588] Registering candidate plugin; module='afsocket', context='source', name='unix-dgram' [2018-10-18T10:15:43.342595] Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram' [2018-10-18T10:15:43.342602] Registering candidate plugin; module='afsocket', context='source', name='tcp' [2018-10-18T10:15:43.342608] Registering candidate plugin; module='afsocket', context='destination', name='tcp' [2018-10-18T10:15:43.342618] Registering candidate plugin; module='afsocket', context='source', name='tcp6' [2018-10-18T10:15:43.342779] Registering candidate plugin; module='afsocket', context='destination', name='tcp6' [2018-10-18T10:15:43.342788] Registering candidate plugin; module='afsocket', context='source', name='udp' [2018-10-18T10:15:43.342794] Registering candidate plugin; module='afsocket', context='destination', name='udp' [2018-10-18T10:15:43.342801] Registering candidate plugin; module='afsocket', context='source', name='udp6' [2018-10-18T10:15:43.342807] Registering candidate plugin; module='afsocket', context='destination', name='udp6' [2018-10-18T10:15:43.342813] Registering candidate plugin; module='afsocket', context='source', name='syslog' [2018-10-18T10:15:43.342820] Registering candidate plugin; module='afsocket', context='destination', name='syslog' [2018-10-18T10:15:43.342830] Registering candidate plugin; module='afsocket', context='source', name='network' [2018-10-18T10:15:43.342837] Registering candidate plugin; module='afsocket', context='destination', name='network' [2018-10-18T10:15:43.342843] Registering candidate plugin; module='afsocket', context='source', name='systemd-syslog' [2018-10-18T10:15:43.342946] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libafstomp.so', module='afstomp' [2018-10-18T10:15:43.343208] Registering candidate plugin; module='afstomp', context='destination', name='stomp' [2018-10-18T10:15:43.343252] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libafuser.so', module='afuser' [2018-10-18T10:15:43.343427] Registering candidate plugin; module='afuser', context='destination', name='usertty' [2018-10-18T10:15:43.343459] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libappmodel.so', module='appmodel' [2018-10-18T10:15:43.343663] Registering candidate plugin; module='appmodel', context='root', name='application' [2018-10-18T10:15:43.343674] Registering candidate plugin; module='appmodel', context='parser', name='app-parser' [2018-10-18T10:15:43.343700] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libbasicfuncs.so', module='basicfuncs' [2018-10-18T10:15:43.343911] Registering candidate plugin; module='basicfuncs', context='template-func', name='grep' [2018-10-18T10:15:43.343921] Registering candidate plugin; module='basicfuncs', context='template-func', name='if' [2018-10-18T10:15:43.343936] Registering candidate plugin; module='basicfuncs', context='template-func', name='or' [2018-10-18T10:15:43.343944] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-lookup' [2018-10-18T10:15:43.343951] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length' [2018-10-18T10:15:43.343957] Registering candidate plugin; module='basicfuncs', context='template-func', name='context-values' [2018-10-18T10:15:43.343964] Registering candidate plugin; module='basicfuncs', context='template-func', name='echo' [2018-10-18T10:15:43.343971] Registering candidate plugin; module='basicfuncs', context='template-func', name='length' [2018-10-18T10:15:43.343977] Registering candidate plugin; module='basicfuncs', context='template-func', name='substr' [2018-10-18T10:15:43.343984] Registering candidate plugin; module='basicfuncs', context='template-func', name='strip' [2018-10-18T10:15:43.343991] Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize' [2018-10-18T10:15:43.343998] Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase' [2018-10-18T10:15:43.344004] Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase' [2018-10-18T10:15:43.344011] Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter' [2018-10-18T10:15:43.344017] Registering candidate plugin; module='basicfuncs', context='template-func', name='padding' [2018-10-18T10:15:43.344024] Registering candidate plugin; module='basicfuncs', context='template-func', name='binary' [2018-10-18T10:15:43.344031] Registering candidate plugin; module='basicfuncs', context='template-func', name='dirname' [2018-10-18T10:15:43.344049] Registering candidate plugin; module='basicfuncs', context='template-func', name='basename' [2018-10-18T10:15:43.344057] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-concat' [2018-10-18T10:15:43.344064] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-head' [2018-10-18T10:15:43.344070] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-nth' [2018-10-18T10:15:43.344077] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-tail' [2018-10-18T10:15:43.344084] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-slice' [2018-10-18T10:15:43.344091] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-count' [2018-10-18T10:15:43.344098] Registering candidate plugin; module='basicfuncs', context='template-func', name='list-append' [2018-10-18T10:15:43.344105] Registering candidate plugin; module='basicfuncs', context='template-func', name='+' [2018-10-18T10:15:43.344111] Registering candidate plugin; module='basicfuncs', context='template-func', name='-' [2018-10-18T10:15:43.344118] Registering candidate plugin; module='basicfuncs', context='template-func', name='*' [2018-10-18T10:15:43.344124] Registering candidate plugin; module='basicfuncs', context='template-func', name='/' [2018-10-18T10:15:43.344131] Registering candidate plugin; module='basicfuncs', context='template-func', name='%' [2018-10-18T10:15:43.344140] Registering candidate plugin; module='basicfuncs', context='template-func', name='sum' [2018-10-18T10:15:43.344147] Registering candidate plugin; module='basicfuncs', context='template-func', name='min' [2018-10-18T10:15:43.344154] Registering candidate plugin; module='basicfuncs', context='template-func', name='max' [2018-10-18T10:15:43.344161] Registering candidate plugin; module='basicfuncs', context='template-func', name='average' [2018-10-18T10:15:43.344167] Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int' [2018-10-18T10:15:43.344174] Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line' [2018-10-18T10:15:43.344181] Registering candidate plugin; module='basicfuncs', context='template-func', name='env' [2018-10-18T10:15:43.344188] Registering candidate plugin; module='basicfuncs', context='template-func', name='template' [2018-10-18T10:15:43.344194] Registering candidate plugin; module='basicfuncs', context='template-func', name='urlencode' [2018-10-18T10:15:43.344230] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libcef.so', module='cef' [2018-10-18T10:15:43.344455] Registering candidate plugin; module='cef', context='template-func', name='format-cef-extension' [2018-10-18T10:15:43.344497] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libconfgen.so', module='confgen' [2018-10-18T10:15:43.344745] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libcryptofuncs.so', module='cryptofuncs' [2018-10-18T10:15:43.344971] Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid' [2018-10-18T10:15:43.344986] Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash' [2018-10-18T10:15:43.344997] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1' [2018-10-18T10:15:43.345091] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256' [2018-10-18T10:15:43.345103] Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512' [2018-10-18T10:15:43.345115] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4' [2018-10-18T10:15:43.345125] Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5' [2018-10-18T10:15:43.345161] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libcsvparser.so', module='csvparser' [2018-10-18T10:15:43.345350] Registering candidate plugin; module='csvparser', context='parser', name='csv-parser' [2018-10-18T10:15:43.345388] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libdate.so', module='date' [2018-10-18T10:15:43.345572] Registering candidate plugin; module='date', context='parser', name='date-parser' [2018-10-18T10:15:43.345603] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libdbparser.so', module='dbparser' [2018-10-18T10:15:43.345819] Registering candidate plugin; module='dbparser', context='parser', name='db-parser' [2018-10-18T10:15:43.345830] Registering candidate plugin; module='dbparser', context='parser', name='grouping-by' [2018-10-18T10:15:43.345859] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libdisk-buffer.so', module='disk-buffer' [2018-10-18T10:15:43.346027] Registering candidate plugin; module='disk-buffer', context='inner-dest', name='disk_buffer' [2018-10-18T10:15:43.346071] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libgraphite.so', module='graphite' [2018-10-18T10:15:43.346224] Registering candidate plugin; module='graphite', context='template-func', name='graphite_output' [2018-10-18T10:15:43.346260] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libjson-plugin.so', module='json-plugin' [2018-10-18T10:15:43.346520] Registering candidate plugin; module='json-plugin', context='parser', name='json-parser' [2018-10-18T10:15:43.346531] Registering candidate plugin; module='json-plugin', context='template-func', name='format_json' [2018-10-18T10:15:43.346571] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libkvformat.so', module='kvformat' [2018-10-18T10:15:43.346744] Registering candidate plugin; module='kvformat', context='parser', name='kv-parser' [2018-10-18T10:15:43.346754] Registering candidate plugin; module='kvformat', context='parser', name='linux-audit-parser' [2018-10-18T10:15:43.346762] Registering candidate plugin; module='kvformat', context='template-func', name='format-welf' [2018-10-18T10:15:43.346789] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='liblinux-kmsg-format.so', module='linux-kmsg-format' [2018-10-18T10:15:43.346936] Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg' [2018-10-18T10:15:43.346963] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libmap-value-pairs.so', module='map-value-pairs' [2018-10-18T10:15:43.347137] Registering candidate plugin; module='map-value-pairs', context='parser', name='map_value_pairs' [2018-10-18T10:15:43.347164] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libpseudofile.so', module='pseudofile' [2018-10-18T10:15:43.347330] Registering candidate plugin; module='pseudofile', context='destination', name='pseudofile' [2018-10-18T10:15:43.347357] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libsdjournal.so', module='sdjournal' [2018-10-18T10:15:43.347538] Registering candidate plugin; module='sdjournal', context='source', name='systemd-journal' [2018-10-18T10:15:43.347569] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libsnmptrapd-parser.so', module='snmptrapd-parser' [2018-10-18T10:15:43.347732] Registering candidate plugin; module='snmptrapd-parser', context='parser', name='snmptrapd-parser' [2018-10-18T10:15:43.347762] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libstardate.so', module='stardate' [2018-10-18T10:15:43.347912] Registering candidate plugin; module='stardate', context='template-func', name='stardate' [2018-10-18T10:15:43.347938] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libsyslogformat.so', module='syslogformat' [2018-10-18T10:15:43.348120] Registering candidate plugin; module='syslogformat', context='format', name='syslog' [2018-10-18T10:15:43.348134] Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser' [2018-10-18T10:15:43.348158] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libsystem-source.so', module='system-source' [2018-10-18T10:15:43.348343] Registering candidate plugin; module='system-source', context='source', name='system' [2018-10-18T10:15:43.348370] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libtags-parser.so', module='tags-parser' [2018-10-18T10:15:43.348594] Registering candidate plugin; module='tags-parser', context='parser', name='tags-parser' [2018-10-18T10:15:43.348635] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libtfgetent.so', module='tfgetent' [2018-10-18T10:15:43.348857] Registering candidate plugin; module='tfgetent', context='template-func', name='getent' [2018-10-18T10:15:43.348898] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libxml.so', module='xml' [2018-10-18T10:15:43.349163] Registering candidate plugin; module='xml', context='parser', name='xml' [2018-10-18T10:15:43.349219] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libmod-java.so', module='mod-java' [2018-10-18T10:15:43.355461] Registering candidate plugin; module='mod-java', context='options', name='jvm_options' [2018-10-18T10:15:43.355486] Registering candidate plugin; module='mod-java', context='destination', name='java' [2018-10-18T10:15:43.355664] Reading shared object for a candidate module; path='//usr/lib64/syslog-ng', fname='libhook-commands.so', module='hook-commands' [2018-10-18T10:15:43.355904] Registering candidate plugin; module='hook-commands', context='inner-dest', name='hook-commands' [2018-10-18T10:15:43.355922] Registering candidate plugin; module='hook-commands', context='inner-src', name='hook-commands' [2018-10-18T10:15:43.356116] Finishing include; filename='/usr/share/syslog-ng/include/scl/apache/apache.conf', depth='2' [2018-10-18T10:15:43.356157] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/cim/adapter.conf', depth='2' [2018-10-18T10:15:43.356529] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.356794] Module loaded and initialized successfully; module='appmodel' [2018-10-18T10:15:43.356897] Finishing include; filename='/usr/share/syslog-ng/include/scl/cim/adapter.conf', depth='2' [2018-10-18T10:15:43.356927] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2018-10-18T10:15:43.357009] Attempted to register the same plugin multiple times, dropping the old one; context='parser', name='json-parser' [2018-10-18T10:15:43.357027] Attempted to register the same plugin multiple times, dropping the old one; context='template-func', name='format_json' [2018-10-18T10:15:43.357057] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.357314] Finishing include; filename='/usr/share/syslog-ng/include/scl/cim/template.conf', depth='2' [2018-10-18T10:15:43.357353] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/cisco/plugin.conf', depth='2' [2018-10-18T10:15:43.357609] Finishing include; filename='/usr/share/syslog-ng/include/scl/cisco/plugin.conf', depth='2' [2018-10-18T10:15:43.357638] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/default-network-drivers/plugin.conf', depth='2' [2018-10-18T10:15:43.357791] Finishing include; filename='/usr/share/syslog-ng/include/scl/default-network-drivers/plugin.conf', depth='2' [2018-10-18T10:15:43.357819] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2018-10-18T10:15:43.357892] Attempted to register the same plugin multiple times, dropping the old one; context='parser', name='json-parser' [2018-10-18T10:15:43.357903] Attempted to register the same plugin multiple times, dropping the old one; context='template-func', name='format_json' [2018-10-18T10:15:43.357909] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.358256] Finishing include; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2' [2018-10-18T10:15:43.358294] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/ewmm/ewmm.conf', depth='2' [2018-10-18T10:15:43.358380] Attempted to register the same plugin multiple times, dropping the old one; context='parser', name='json-parser' [2018-10-18T10:15:43.358396] Attempted to register the same plugin multiple times, dropping the old one; context='template-func', name='format_json' [2018-10-18T10:15:43.358403] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.358883] Finishing include; filename='/usr/share/syslog-ng/include/scl/ewmm/ewmm.conf', depth='2' [2018-10-18T10:15:43.358919] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2018-10-18T10:15:43.359012] Finishing include; filename='/usr/share/syslog-ng/include/scl/graphite/plugin.conf', depth='2' [2018-10-18T10:15:43.359059] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/graylog2/plugin.conf', depth='2' [2018-10-18T10:15:43.359144] Attempted to register the same plugin multiple times, dropping the old one; context='parser', name='json-parser' [2018-10-18T10:15:43.359159] Attempted to register the same plugin multiple times, dropping the old one; context='template-func', name='format_json' [2018-10-18T10:15:43.359171] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.359557] Module loaded and initialized successfully; module='basicfuncs' [2018-10-18T10:15:43.359634] Finishing include; filename='/usr/share/syslog-ng/include/scl/graylog2/plugin.conf', depth='2' [2018-10-18T10:15:43.359664] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2018-10-18T10:15:43.359811] Finishing include; filename='/usr/share/syslog-ng/include/scl/hdfs/plugin.conf', depth='2' [2018-10-18T10:15:43.359842] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/iptables/iptables.conf', depth='2' [2018-10-18T10:15:43.360013] Finishing include; filename='/usr/share/syslog-ng/include/scl/iptables/iptables.conf', depth='2' [2018-10-18T10:15:43.360061] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2018-10-18T10:15:43.360177] Finishing include; filename='/usr/share/syslog-ng/include/scl/kafka/plugin.conf', depth='2' [2018-10-18T10:15:43.360205] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/loadbalancer/plugin.conf', depth='2' [2018-10-18T10:15:43.360588] Module loaded and initialized successfully; module='confgen' [2018-10-18T10:15:43.360629] Finishing include; filename='/usr/share/syslog-ng/include/scl/loadbalancer/plugin.conf', depth='2' [2018-10-18T10:15:43.360666] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2018-10-18T10:15:43.360784] Attempted to register the same plugin multiple times, dropping the old one; context='parser', name='json-parser' [2018-10-18T10:15:43.360801] Attempted to register the same plugin multiple times, dropping the old one; context='template-func', name='format_json' [2018-10-18T10:15:43.360814] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.360881] Finishing include; filename='/usr/share/syslog-ng/include/scl/loggly/loggly.conf', depth='2' [2018-10-18T10:15:43.360911] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2018-10-18T10:15:43.361098] Attempted to register the same plugin multiple times, dropping the old one; context='parser', name='json-parser' [2018-10-18T10:15:43.361110] Attempted to register the same plugin multiple times, dropping the old one; context='template-func', name='format_json' [2018-10-18T10:15:43.361117] Module loaded and initialized successfully; module='json-plugin' [2018-10-18T10:15:43.361173] Finishing include; filename='/usr/share/syslog-ng/include/scl/logmatic/logmatic.conf', depth='2' [2018-10-18T10:15:43.361214] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2018-10-18T10:15:43.361315] Finishing include; filename='/usr/share/syslog-ng/include/scl/mbox/mbox.conf', depth='2' [2018-10-18T10:15:43.361343] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2018-10-18T10:15:43.361441] Finishing include; filename='/usr/share/syslog-ng/include/scl/nodejs/plugin.conf', depth='2' [2018-10-18T10:15:43.361467] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/osquery/plugin.conf', depth='2' [2018-10-18T10:15:43.361655] Finishing include; filename='/usr/share/syslog-ng/include/scl/osquery/plugin.conf', depth='2' [2018-10-18T10:15:43.361695] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2018-10-18T10:15:43.361808] Finishing include; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2' [2018-10-18T10:15:43.361836] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2018-10-18T10:15:43.361914] Global value changed; define='balabit.credit-card-regexp', value='(:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35d{3})d{11})' [2018-10-18T10:15:43.361990] Finishing include; filename='/usr/share/syslog-ng/include/scl/rewrite/cc-mask.conf', depth='2' [2018-10-18T10:15:43.362030] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/snmptrap/snmptrapd-source.conf', depth='2' [2018-10-18T10:15:43.362195] Finishing include; filename='/usr/share/syslog-ng/include/scl/snmptrap/snmptrapd-source.conf', depth='2' [2018-10-18T10:15:43.362250] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2018-10-18T10:15:43.362336] Finishing include; filename='/usr/share/syslog-ng/include/scl/solaris/plugin.conf', depth='2' [2018-10-18T10:15:43.362363] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/sudo/sudo.conf', depth='2' [2018-10-18T10:15:43.362474] Finishing include; filename='/usr/share/syslog-ng/include/scl/sudo/sudo.conf', depth='2' [2018-10-18T10:15:43.362501] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2018-10-18T10:15:43.362600] Module loaded and initialized successfully; module='confgen' [2018-10-18T10:15:43.362618] Finishing include; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2' [2018-10-18T10:15:43.362643] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2018-10-18T10:15:43.362705] Finishing include; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2' [2018-10-18T10:15:43.362728] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/telegram/telegram.conf', depth='2' [2018-10-18T10:15:43.362810] Plugin module not found in 'module-path'; module-path='//usr/lib64/syslog-ng', module='http' [2018-10-18T10:15:43.362822] Included file was skipped because of a missing module; module='http', location='/usr/share/syslog-ng/include/scl/telegram/telegram.conf:23:11' [2018-10-18T10:15:43.362829] Finishing include; filename='/usr/share/syslog-ng/include/scl/telegram/telegram.conf', depth='2' [2018-10-18T10:15:43.362854] Starting to read include file; filename='/usr/share/syslog-ng/include/scl/windowseventlog/plugin.conf', depth='2' [2018-10-18T10:15:43.362924] Finishing include; filename='/usr/share/syslog-ng/include/scl/windowseventlog/plugin.conf', depth='2' [2018-10-18T10:15:43.362958] Global value changed; define='java-module-dir', value='//usr/lib64/syslog-ng/java-modules' [2018-10-18T10:15:43.362975] Finishing include; filename='/etc/syslog-ng/scl.conf', depth='1' [2018-10-18T10:15:43.363581] Module loaded and initialized successfully; module='afsocket' [2018-10-18T10:15:43.364134] Module loaded and initialized successfully; module='tags-parser' [2018-10-18T10:15:43.364190] Finishing include; content='block parser ewmm-parser() at /usr/share/syslog-ng/include/scl/ewmm/ewmm.conf:40', depth='2' [2018-10-18T10:15:43.364207] Finishing include; content='block source ewmm() at /usr/share/syslog-ng/include/scl/ewmm/ewmm.conf:92', depth='1' [2018-10-18T10:15:43.364266] Adding include file; filename='/etc/syslog-ng/conf.d/es.conf', depth='1' [2018-10-18T10:15:43.364281] Starting to read include file; filename='/etc/syslog-ng/conf.d/es.conf', depth='1' [2018-10-18T10:15:43.366863] Module loaded and initialized successfully; module='mod-java' [2018-10-18T10:15:43.367093] Finishing include; content='block destination elasticsearch2() at /usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf:59', depth='2' [2018-10-18T10:15:43.367153] Finishing include; filename='/etc/syslog-ng/conf.d/es.conf', depth='1' [2018-10-18T10:15:43.367376] Error creating persistent state file; filename='/var/lib/syslog-ng/syslog-ng.persist-', error='Permission denied (13)'

I am looking forward to here from you soon.

Rgards, Jawad

JawadKM commented 6 years ago

and when I go "Dir" I can see the syslog-ng.persist

cd /var/lib/syslog-ng [user@open syslog-ng]$ ll total 16 srwxr-xr-x. 1 root root 0 Oct 18 10:31 syslog-ng.ctl -rw-------. 1 root root 16384 Oct 18 10:31 syslog-ng.persist

gaborznagy commented 6 years ago

@JawadKM: Can you help us narrow down where is exactly any problem?

  1. Last comment was about a persist file creation problem, but I see a later comment shows that the persist file is created, so I guess you have found the cause

  2. as the issue name suggests you have problems with elasticssearch destination, though your statistics shows that it's working, though you have a lot of dropped messages:

    dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;written;5000
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;dropped;2541
JawadKM commented 6 years ago

@gaborznagy Thanks for the reply. So could you please guide how how to fix, what you have stated in your first and second point. I need your help, as this is my first time working with syslog-ng.

Thanks, Jawad

JawadKM commented 6 years ago

@gaborznagy Also I have thise error in "-Fdev" result syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted'

JawadKM commented 6 years ago

@gaborznagy and when I tried addressing the below issue [2018-10-18T10:15:43.367376] Error creating persistent state file; filename='/var/lib/syslog-ng/syslog-ng.persist-', error='Permission denied (13)'

it started giving me these errors

[2018-10-19T11:29:32.399183] WARNING: With use-dns(no), dns-cache() will be forced to 'no' too!; [2018-10-19T11:29:32.399438] Module loaded and initialized successfully; module='syslogformat' [2018-10-19T11:29:32.400471] Error binding socket; addr='AF_INET(0.0.0.0:514)', error='Permission denied (13)' [2018-10-19T11:29:32.400498] Error initializing message pipeline; plugin_name='network', location='block source ewmm() at /usr/share/syslog-ng/include/scl/ewmm/ewmm.conf:92:7:25'

JawadKM commented 6 years ago

@szemere @gaborznagy @faxm0dem I uninstalled and re installed elasticsearch, deleted the index from kibana and added it again, and started getting the latest logs. this resulted in losing all the previous log, but I can live with that as this is not Prod. Now the issue is the drop messages as indicated by @gaborznagy in his above comment. Also with the new install I am getting a lot of drop messages, would you be able to assist me with this issue. I believe this issue was caused by huge number of dropped messages.

$ sudo syslog-ng-ctl stats

SourceName;SourceId;SourceInstance;State;Type;Number source;s_network;;a;processed;545 destination;d_elastic;;a;processed;545 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;dropped;37 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;processed;545 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;queued;0 dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;written;508 center;;received;a;processed;545 center;;queued;a;processed;545 global;scratch_buffers_count;;a;queued;167503724576 global;payload_reallocs;;a;processed;4 global;sdata_updates;;a;processed;545 global;scratch_buffers_bytes;;a;queued;0 global;msg_clones;;a;processed;0

looking forward to here from you soon.

szemere commented 5 years ago

Dear @JawadKM, sorry for the late answer, but I have to train myself too in the topic. I try to summarize my current progress, hope it will help you along the way:

@version:3.17
@include "scl.conf"

options {
    flush_lines(0);
    time_reopen (10);
    log_fifo_size (1000);
    chain_hostnames (off);
    use_dns (no);
    use_fqdn (no);
    create_dirs (no);
    keep_hostname (yes);
};

source s_file {
    file("/source/input.log");
};

destination d_elastic {
    elasticsearch2 (
        cluster("es-syslog-ng")
        client-mode("http")
        index("syslog-ng")
        type("test")
        template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
        port("9200")
        server("192.168.33.10")

    )
};

log {
    #source(s_network);
    source(s_file);
    destination(d_elastic);
    flags(flow-control);
};

https://github.com/appbaseio/dejavu#cors

http.port: 9200
http.cors.allow-origin: "http://localhost:1358"
http.cors.enabled: true
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true

Here is some consetutive syslog-ng-ctl stats output:

SourceName;SourceId;SourceInstance;State;Type;Number
destination;d_elastic;;a;processed;487307
center;;received;a;processed;487307
center;;queued;a;processed;487307
global;scratch_buffers_count;;a;queued;21474836505
source;s_file;;a;processed;26
global;payload_reallocs;;a;processed;0
global;sdata_updates;;a;processed;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;dropped;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;processed;487307
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;queued;100
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;written;487207
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;0
source;s_net;;a;processed;487281
SourceName;SourceId;SourceInstance;State;Type;Number
destination;d_elastic;;a;processed;726670
center;;received;a;processed;726670
center;;queued;a;processed;726670
global;scratch_buffers_count;;a;queued;21474836505
source;s_file;;a;processed;26
global;payload_reallocs;;a;processed;0
global;sdata_updates;;a;processed;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;dropped;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;processed;726670
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;queued;100
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;written;726570
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;0
source;s_net;;a;processed;726644

Note: my syslog-ng was processing messages waaay after I terminated the yes | ... command. Since the yes packets was so small, my kernel and syslog-ng stored several hundreds in the incoming buffer.

SourceName;SourceId;SourceInstance;State;Type;Number
destination;d_elastic;;a;processed;1763585
center;;received;a;processed;1763585
center;;queued;a;processed;1763585
global;scratch_buffers_count;;a;queued;21474836505
source;s_file;;a;processed;26
global;payload_reallocs;;a;processed;0
global;sdata_updates;;a;processed;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;dropped;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;processed;1763585
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;queued;99
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;written;1763485
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;0
source;s_net;;a;processed;1763559

After syslog-ng stopped running:

SourceName;SourceId;SourceInstance;State;Type;Number
destination;d_elastic;;a;processed;2455877
center;;received;a;processed;2455877
center;;queued;a;processed;2455877
global;scratch_buffers_count;;a;queued;21474836505
source;s_file;;a;processed;26
global;payload_reallocs;;a;processed;0
global;sdata_updates;;a;processed;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;dropped;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;processed;2455877
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;queued;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,es-syslog-ng,syslog-ng;a;written;2455877
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;0
source;s_net;;a;processed;2455851

I dont wan't to attach images to this comment, but the dejavu web interface showed me the following information in the header: Showing 20 of total 2455851 which is exactly the same as the processed messages by syslog-ng. (A better example would be to count the inputs as well, but since your issue is between the syslog-ng destination and elasticsearch, I ignored the source part.)

In the previous comments I can see several different issues beside elasticsearch. i.e.: persist file permission problems. If it is suitable for you, I would prefer to handle them separatedly from the elasticsearch part. In the following steps we could work on to figure out the differences between your setup, and the described, controlled environment.

Br, Laci