search

fluent-plugins-nursery / fluent-plugin-concat

Fluentd Filter plugin to concatenate multiline log separated in multiple events.
MIT License
108 stars 33 forks source link

log with lengthy exception is getting dropped #90

Open NareshAnaparthi opened 4 years ago

NareshAnaparthi commented 4 years ago

Problem

Trying to concatenate Multi line records into single log record Working well for all kinds of exceptions/startup logs Lengthy exceptions are getting dropped. ...

Steps to replicate

Please tryout below config and seeing lengthy lines/exceptions are getting dropped. Would like to know if we can limit lines or increase if its related to Buffer limitation

Provide example config and message

2020-04-03T02:57:17.605Z ERROR

[Server] FATAL error: com.vmware.vdi.pop.configuration.ConfigurationException$IOConfigurationException: Failed to read configuration com.vmware.vdi.pop.configuration.ConfigurationException$IOConfigurationException: Failed to read configuration at com.vmware.vdi.pop.configuration.ConfigurationManager.read(ConfigurationManager.java:74) at com.vmware.vdi.pop.configuration.ConfigUtils.readConfig(ConfigUtils.java:88) at com.vmware.vdi.hydra.connector.server.Server.main(Server.java:287) Caused by: java.io.FileNotFoundException: ./tls/connector-api.config (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070) at com.vmware.vdi.pop.configuration.ConfigurationManager.read(ConfigurationManager.java:68) ... 2 common frames omitted Native Memory Tracking: Total: reserved=2315290KB, committed=68470KB

  • Java Heap (reserved=983040KB, committed=32768KB) (mmap: reserved=983040KB, committed=32768KB)
  • Class (reserved=1063273KB, committed=16489KB) (classes #2827) (malloc=361KB #1208) (mmap: reserved=1062912KB, committed=16128KB)
  • Thread (reserved=5718KB, committed=5718KB) (thread #10) (stack: reserved=5672KB, committed=5672KB) (malloc=33KB #52) (arena=13KB #19)
  • Code (reserved=250014KB, committed=3342KB) (malloc=414KB #1119) (mmap: reserved=249600KB, committed=2928KB)
  • GC (reserved=3215KB, committed=123KB) (malloc=7KB #84) (mmap: reserved=3208KB, committed=116KB)
  • Compiler (reserved=138KB, committed=138KB) (malloc=7KB #90) (arena=131KB #5)
  • Internal (reserved=506KB, committed=506KB) (malloc=474KB #3772) (mmap: reserved=32KB, committed=32KB)
  • Symbol (reserved=4910KB, committed=4910KB) (malloc=3815KB #30989) (arena=1095KB #1)
  • Native Memory Tracking (reserved=589KB, committed=589KB) (malloc=3KB #35) (tracking overhead=586KB)
  • Arena Chunk (reserved=3887KB, committed=3887KB) (malloc=3887KB)_

Expected Behavior

concat plugin should be able concat above multiline exception/message into single log record. OR Need a way to truncate instead of dropping entire message. ...

Your environment

Concat Plugin Config <filter **> @type concat key log multiline_start_regexp /^\d{4}[-/]\d{1,2}[-/]\d{1,2}/ separator "" flush_interval 5

  • OS version Linux dev1-westus2-jumpbox 5.0.0-1028-azure #30~18.04.1-Ubuntu SMP Fri Dec 6 11:47:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • paste result of fluentd --version or td-agent --version

  • plugin version

    • paste boot log of fluentd or td-agent
    • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
kaisermario commented 4 years ago

Any updates on this? Having also issues with concatenation of multineline logs and multiline_start_regexp.

okkez commented 4 years ago

Hmm, I cannot reproduce the issue.