search

sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint
https://sematext.com/logagent
Apache License 2.0
389 stars 79 forks source link

output-gelf plugin uses up all ports and crashes #221

Closed hkclark closed 4 years ago

hkclark commented 4 years ago

We have been trying to use the output-gelf plugin and have noticed after 1-2 hours we get these exceptions and logagent stops working:

...
logagent_1        | 2020-04-15T12:23:05.023Z v3.0.31 pid[6] Logagent report: pid[6] 60000 ms 146 lines parsed.  2 lines/s 0.000 MB/s - empty lines: 0
logagent_1        | 2020-04-15T12:23:05.024Z v3.0.31 pid[6] Logagent stats: tokens.used=0 logs.shipped=0 http.failed=0 http.retranstmit=0 parser.throughput.lps=2 parser.throughput.bps=239 parser.lines.parsed=146
logagent_1        | 2020-04-15T12:23:05.024Z v3.0.31 pid[6] Memory stats:  heapUsedMB=114 heapTotalMB=115 memoryRssMB=176
logagent_1        | 2020-04-15T12:24:05.023Z v3.0.31 pid[6] Logagent report: pid[6] 60001 ms 200 lines parsed.  3 lines/s 0.000 MB/s - empty lines: 0
logagent_1        | 2020-04-15T12:24:05.024Z v3.0.31 pid[6] Logagent stats: tokens.used=0 logs.shipped=0 http.failed=0 http.retranstmit=0 parser.throughput.lps=3 parser.throughput.bps=308 parser.lines.parsed=200
logagent_1        | 2020-04-15T12:24:05.024Z v3.0.31 pid[6] Memory stats:  heapUsedMB=108 heapTotalMB=110 memoryRssMB=172
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: getaddrinfo EADDRINUSE graylog
logagent_1        |   Error: getaddrinfo EADDRINUSE graylog
logagent_1        |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: getaddrinfo EADDRINUSE graylog
logagent_1        |   Error: getaddrinfo EADDRINUSE graylog
logagent_1        |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
logagent_1        | Error: getaddrinfo EADDRINUSE graylog
logagent_1        |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26) {
logagent_1        |   errno: 'EADDRINUSE',
logagent_1        |   code: 'EADDRINUSE',
logagent_1        |   syscall: 'getaddrinfo',
logagent_1        |   hostname: 'graylog'
logagent_1        | }
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: bind EADDRINUSE 0.0.0.0
logagent_1        |   Error: bind EADDRINUSE 0.0.0.0
logagent_1        |     at dgram.js:339:20
logagent_1        |     at processTicksAndRejections (internal/process/task_queues.js:85:21)
logagent_1        |     at runNextTicks (internal/process/task_queues.js:66:3)
logagent_1        |     at processImmediate (internal/timers.js:429:9)
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: bind EADDRINUSE 0.0.0.0
logagent_1        |   Error: bind EADDRINUSE 0.0.0.0
logagent_1        |     at dgram.js:339:20
logagent_1        |     at processTicksAndRejections (internal/process/task_queues.js:85:21)
logagent_1        |     at runNextTicks (internal/process/task_queues.js:66:3)
logagent_1        |     at processImmediate (internal/timers.js:429:9)
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: bind EADDRINUSE 0.0.0.0
logagent_1        |   Error: bind EADDRINUSE 0.0.0.0
logagent_1        |     at dgram.js:339:20
logagent_1        |     at processTicksAndRejections (internal/process/task_queues.js:85:21)
logagent_1        |     at runNextTicks (internal/process/task_queues.js:66:3)
logagent_1        |     at processImmediate (internal/timers.js:429:9)
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: bind EADDRINUSE 0.0.0.0
logagent_1        |   Error: bind EADDRINUSE 0.0.0.0
logagent_1        |     at dgram.js:339:20
logagent_1        |     at processTicksAndRejections (internal/process/task_queues.js:85:21)
logagent_1        |     at runNextTicks (internal/process/task_queues.js:66:3)
logagent_1        |     at processImmediate (internal/timers.js:429:9)
logagent_1        | Please contact support@sematext.com to report the error:
logagent_1        | UncaughtException:Error: bind EADDRINUSE 0.0.0.0
logagent_1        |   Error: bind EADDRINUSE 0.0.0.0
logagent_1        |     at dgram.js:339:20
logagent_1        |     at processTicksAndRejections (internal/process/task_queues.js:85:21)
logagent_1        |     at runNextTicks (internal/process/task_queues.js:66:3)
...

We are running this in the standard Docker Hub container (sematext/logagent:latest) where we have a second stage in the build process to have it npm install -g --unsafe-perm graygelf to satisfy the requirement for graygelf, but otherwise it's the standard container.

I have noticed if I docker-compose exec to the container and watch netstat -tupan that the number of UDP ports in use steadily grows until the exceptions start happening.

otisg commented 4 years ago

@hkclark I wonder if the problem is in 'graygelf' or in whatever is running on that graylog. Can you try temporarily shipping your logs to Sematext to see if you still hit this problem?

hkclark commented 4 years ago

@oba11 Thanks for the reply. Yes, graygelf is being used... I'm installing that into the standard logagent Docker container with npm. Is the suggestion to send to Sematext using the default:

output:
  logsene:
    module: elasticsearch
    url: $LOGSENE_RECEIVER_URL
    ...

Or to use the GELF output module to send to Sematext (if so I didn't know that was an option). However, if I switch to logsene/elasticsearch I won't get this issue -- only only when I'm using the "output-gelf" plugin.

otisg commented 4 years ago

I think you can ship GELF logs to the "socket receiver" for JSON listed at https://sematext.com/docs/logs/sending-log-events/

hkclark commented 4 years ago

@otisg I just submitted a Pull Request that fixes it for me. I think the graygelf object should be created once in the constructor vs creating one every time the eventHandler fires. As I mentioned in the PR, let me know if I can do anything to assist. Thanks!

hkclark commented 4 years ago

See PR #222

adnanrahic commented 4 years ago

Closing this as it was fixed in #222.

adnanrahic commented 4 years ago

@hkclark Hey! I've released 3.0.32 with this fix. Feel free to try it out!

hkclark commented 4 years ago

Thanks! Also saw the update on Dockerhub and we are pulling that image. Thanks again for all the help!

adnanrahic commented 4 years ago

Awesome! Pleasure was all mine.