categraf收集应用异常信息时将多行数据存入到一条kafka的message中

jiayjlr commented 4 weeks ago

Relevant config.toml

logs.toml

[logs]
api_key = "ef4ahfbwzwwtlwfpbertgq1i6mq0ab1q"
enable = true
send_to = "10.2.139.37:9092,10.2.139.37:9093,10.2.139.37:9094"
send_type = "kafka"
topic = "flashcatcloud"
use_compress = false
send_with_tls = false
batch_wait = 5
run_path = "/usr/local/categraf/conf/run"
open_files_limit = 100
scan_period = 10 
frame_size = 9000
pipeline=8
chan_size=10000
batch_max_concurrence=100
batch_max_size=10000
batch_max_content_size=1000000
collect_container_all = true
  [[logs.items]]
  type = "file"
  path = "/data/logs/wukong/wukong-test.log"
  source = "wukongcategraf"
  service = "wukongcategraf"
  tags = ["monitoring=wukongcategraf", "env=test"] 
  topic = "wukongcategraf"
    [[logs.items.log_processing_rules]]
    type = "multi_line"
    name = "new_line_with_date"
    pattern="\\d{4}-\\d{2}-\\d{2}"

Logs from categraf

none

System info

v0.3.77-7d33cbabf42b09732ac6bea08cad6261581144ec

Docker

No response

Steps to reproduce

1、使用categraf日志组件将应用的日志记录到kafka中 2、categraf中logs.toml组建配置文件如上 3、执行应用请求抛出异常信息，参考附件：appexception.log文件 appexception.log 4、在categraf的logs.toml配置收集多行信息，配置如下 [[logs.items.log_processing_rules]] type = "multi_line" name = "new_line_with_date" pattern="\d{4}-\d{2}-\d{2}" 5、但是从kafka中看到的消息数据，categraf将多行报错信息写入到同一条消息中 kafka_message.log

应用日志中：appexception.log [2024-10-31 14:11:11.391] 记录的exception异常信息，在[2024-10-31 14:11:11.819]这个时间点也记录了异常信息 kafka消息日志：kafka_message.log 而在kafka中，将[2024-10-31 14:11:11.391] 、[2024-10-31 14:11:11.819]、[2024-10-31 14:11:12.674]都记录到了kafka中的一条消息中

Expected behavior

应用日志中：appexception.log [2024-10-31 14:11:11.391] 记录的exception异常信息，在[2024-10-31 14:11:11.819]这个时间点也记录了异常信息 kafka消息日志：kafka_message.log 而在kafka中，需要将[2024-10-31 14:11:11.391] 、[2024-10-31 14:11:11.819]、[2024-10-31 14:11:12.674]按照时间分别记录到kafka中，每个时间点写入一条消息

Actual behavior

应用日志中：appexception.log [2024-10-31 14:11:11.391] 记录的exception异常信息，在[2024-10-31 14:11:11.819]这个时间点也记录了异常信息 kafka消息日志：kafka_message.log 而在kafka中，将[2024-10-31 14:11:11.391] 、[2024-10-31 14:11:11.819]、[2024-10-31 14:11:12.674]都记录到了kafka中的一条消息中

Additional info

No response

kongfei605 commented 4 weeks ago

https://flashcat.cloud/docs/content/flashcat-monitor/categraf/4-logs-agent/ 带着中括号的这种写成 pattern="\[\d{4}-\d{2}-\d{2}"

jiayjlr commented 4 weeks ago

https://flashcat.cloud/docs/content/flashcat-monitor/categraf/4-logs-agent/ 带着中括号的这种写成 pattern="\[\d{4}-\d{2}-\d{2}"

按照这种配置写法，categraf服务启动失败

jiayjlr commented 4 weeks ago

我这边同时安装了一个filebeat，配置filebeat的pattern格式： multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}' 我这边是否可以参考filebeat的配置在categraf中配置pattern格式为：[[0-9]{4}-[0-9]{2}-[0-9]{2}

jiayjlr commented 4 weeks ago

我这边同时安装了一个filebeat，配置filebeat的pattern格式： multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}' 我这边是否可以参考filebeat的配置在categraf中配置pattern格式为：[[0-9]{4}-[0-9]{2}-[0-9]{2}

使用这种格式进行测试的时候，有时候也会将两条日志内容存放到一条kafka的消息中

jiayjlr commented 4 weeks ago

使用这中pattern的格式能解决我这边的问题 pattern=" \[[0-9]{4}-[0-9]{2}-[0-9]{2}" 在[]前面有两个\

flashcatcloud / categraf