search

sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint
https://sematext.com/logagent
Apache License 2.0
389 stars 79 forks source link

Improve docker socket connection recovery #147

Closed otisg closed 5 years ago

otisg commented 5 years ago

e.g. even when Docker socket connection is dropped.

megastef commented 5 years ago

Is there any specific problem? The container should be deployed with --restart always options and makes a graceful shutdown when docker daemon gets restarted. See logs of a Docker daemon restart below: See https://sematext.com/docs/logagent/installation-docker/


2018-12-17T21:14:17.258Z pid[7] Logagent report: pid[7] 31007 ms 3108 lines parsed.  100 lines/s 0.013 MB/s - empty lines: 0
2018-12-17T21:14:17.258Z pid[7] Logagent stats: usedTokens=3 shippedLogs=3000 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=100
2018-12-17T21:14:17.258Z pid[7] Memory stats:  heapUsedMB=27 heapTotalMB=49 memoryRssMB=143
2018-12-17T21:14:17.263Z pid[7] send xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/logs
2018-12-17T21:14:17.264Z pid[7] send xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/access_common
2018-12-17T21:14:17.265Z pid[7] send xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/access_log_combined
2018-12-17T21:14:19.080Z pid[7] flushed 768 logs for https://logsene-receiver.sematext.com xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/access_log_combined, logs in buffer: 0
Preparing environment...

Content of /etc/sematext/logagent.conf:
options:
  printStats: 60
  suppress: true
  geoipEnabled: true
  diskBufferDir: /tmp/sematext-logagent
parser:
  patternFiles:
    - /opt/logagent/patterns.yml
    - /etc/logagent/patterns.yml
output:
  logsene:
    module: elasticsearch
    url: https://logsene-receiver.sematext.com
    index: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx
/usr/local/bin/logagent -c /etc/sematext/logagent.conf --docker /var/run/docker.sock 
2018-12-17T21:14:58.481Z pid[7] init filter: inputFilter
2018-12-17T21:14:58.496Z pid[7] init filter: outputFilter
2018-12-17T21:14:58.505Z pid[7] load outputFilter: 0 docker-enrichment
2018-12-17T21:14:58.506Z pid[7] init plugins
2018-12-17T21:14:58.507Z pid[7] ../lib/plugins/input/stdin
2018-12-17T21:14:58.639Z pid[7] ../lib/plugins/output/stdout
2018-12-17T21:14:58.696Z pid[7] ../lib/plugins/output/elasticsearch.js
2018-12-17T21:14:59.975Z pid[7] docker-logs
2018-12-17T21:15:01.092Z ignoring pattern file /etc/logagent/patterns.yml Error: ENOENT: no such file or directory, open '/etc/logagent/patterns.yml'
2018-12-17T21:15:01.104Z error watching pattern file:Error: ENOENT: no such file or directory, watch '/etc/logagent/patterns.yml'
2018-12-17T21:15:05.432Z pid[7] Logagent initialized
2018-12-17T21:15:05.640Z pid[7] Container 3dc797b83220adcd78e3e62ccf1de1283ea75b3daa34a737b67edb4997585b12 /sematext-agent setting LOGSENE_ENABLED=true
2018-12-17T21:15:07.153Z pid[7] Logagent report: pid[7] 9848 ms 0 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2018-12-17T21:15:07.153Z pid[7] Logagent stats: usedTokens=0 shippedLogs=0 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0
2018-12-17T21:15:07.154Z pid[7] Memory stats:  heapUsedMB=12 heapTotalMB=16 memoryRssMB=101```