Log cannot format correct

SkyCTing commented 1 month ago

Bug Report

Describe the bug I have more than one servers，I need to collect logs from each server by fluent bit and sent it to fluentd,then insert to mysql and save it to file as a copy.but i failed.

To Reproduce

my original log is like this:

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

Steps to reproduce the problem: my config file of fluent bit is like this
```
[SERVICE]
Flush         5
Daemon        Off
Log_Level     info
```

[INPUT] Name tail Path /data/visit-log*.log Tag visit_log Parser json

[OUTPUT] Name forward Match visit_log Host secret Port secret

my config file of fluentd is like this

@type forward port secret

@type record_transformer enable_ruby @type file path /data/log/visit_log/ @type json

everything is ok now ,but the log i got in fluentd server is like this

{"log":"{\"message\":\"visit log\",\"context\":{\"user_id\":70,\"channel_id\":\"73\",\"click_id\":\"20240911254e89n1ad7a\",\"nm_id\":\"36451\",\"cm_id\":\"2107671\",\"am_id\":\"28737\",\"app_id\":1,\"ip_addr\":\"213.37.122.34\",\"tracking_code\":\"237d11s78f7i5\",\"ua\":\"Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1\",\"device_type\":1,\"url\":\"https://www.google.com",\"date\":\"2024-09-11\",\"hostname\":\"e29df85ad33a\"},\"level\":200,\"level_name\":\"INFO\",\"channel\":\"production\",\"datetime\":\"2024-09-11 14:53:45\",\"extra\":{}}"}


**Expected behavior**
<!--- A clear and concise description of what you expected to happen. -->
I hope  the result may like this

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

or like this

{"log":{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}}


I cannot read the log now ,the content inside is not a valid json,I read the document the make the Parser as json,it does not work

help me ,thank you every one.
**Screenshots**
<!--- If applicable, add screenshots to help explain your problem. -->

**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used:
the version of flurned is :fluent-package 5.0.4 fluentd 1.16.5
the version of fluent bit is:Fluent Bit v3.1.7
* Configuration:
* Environment name and version (e.g. Kubernetes? What version?):
* Server type and version:
* Operating System and version:
* Filters and plugins:

**Additional context**
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

patrick-stephens commented 1 month ago

I simplified the original problem I think, it looks like the JSON parser is being ignored and injecting into log key instead of parsing.

test.log

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

Config and output

$ cat fluent-bit.conf
[INPUT]
  Name Tail
  Path /logs/*
  Read_from_head true
  Parser json

[OUTPUT]
  Name stdout
  Match *

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro -v $PWD/test.log:/logs/test.log:ro fluent/fluent-bit:3.1.7
...
[2024/09/11 09:27:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5 watch_fd=1 name=/logs/test.log
[0] tail.0: [[1726046835.693655839, {}], {"log"=>"{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}"}]
[1] tail.0: [[1726046835.693658048, {}], {"log"=>""}]

patrick-stephens commented 1 month ago

Sub-note the JSON seems to include numbers both as strings and numerals plus an empty structure.

SkyCTing commented 1 month ago

I simplified the original problem I think, it looks like the JSON parser is being ignored and injecting into log key instead of parsing.

test.log

{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}

Config and output

$ cat fluent-bit.conf
[INPUT]
  Name Tail
  Path /logs/*
  Read_from_head true
  Parser json

[OUTPUT]
  Name stdout
  Match *

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro -v $PWD/test.log:/logs/test.log:ro fluent/fluent-bit:3.1.7
...
[2024/09/11 09:27:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5 watch_fd=1 name=/logs/test.log
[0] tail.0: [[1726046835.693655839, {}], {"log"=>"{"message":"visit log","context":{"user_id":70,"channel_id":"73","click_id":"202409112012ebl19934","nm_id":"23654","cm_id":"2099450","am_id":"20279","app_id":1,"ip_addr":"89.135.131.81","tracking_code":"2kelb7b78ehj5","ua":"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36","device_type":4,"url":"https://www.google.com","date":"2024-09-11","hostname":"e29df85ad33a"},"level":200,"level_name":"INFO","channel":"production","datetime":"2024-09-11 13:55:00","extra":{}}"}]
[1] tail.0: [[1726046835.693658048, {}], {"log"=>""}]

Yes，the key problem is JSON parser does not work,even the message is like {"message":"visit log"}

fluent / fluent-bit