jbrownsc
commented
6 years ago Just to piggyback on this.
Using a docker image: fluent/fluent-bit-0.13-dev:0.10 in Kubernetes with the following config
[SERVICE]
Flush 1
Daemon Off
Log_Level debug
Parsers_File parsers.conf
[INPUT]
Name tail
Path /tmp/var/log/containers/test-fluent-69d87bd8bc-65z8d_test-namespace_test-fluent-019f9c0c33d24470d943a399a2defb63b9bbed982b232941043963475067c315.log
Tag kube.*
Parser docker
Refresh_Interval 5
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_JSON_Log On
[OUTPUT]
Name es
Match *
Host [Elastic Search host]
Port 9200
Retry_Limit False
Generate_ID TrueAnd the following log messages
{"log":"{\"json_string_key\": \"json_string_value\"}\n","stream":"stdout","time":"2018-03-09T06:17:20.701547477Z"}
{"log":"STRING\n","stream":"stdout","time":"2018-03-09T06:17:22.104768562Z"}The log message from Line 1 will be indexed as mapped object with nested properties.
(k8s-uw1-gcp/default) ➜ /tmp curl -sX GET http://localhost:9200/fluent-bit/ | jq -r '."fluent-bit".mappings.flb_type.properties.log'
{
"properties": {
"json_string_key": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}The log message from Line 2 will error out as it is cast as a string, and a string can't be injected into a mapped object.
[2018/03/12 22:29:20] [ warn] [out_es] Elasticsearch error
{"took":43,"errors":true,"items":[{"index":{"_index":"fluent-bit","_type":"flb_type","_id":"2b524b83-f7a4-366d-b0c2-ca9ffed44173","_version":4,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3,"_primary_term":1,"status":200}},{"index":{"_index":"fluent-bit","_type":"flb_type","_id":"498f7d28-1630-9083-d1e2-85a66258540f","status":400,"error":{"type":"mapper_parsing_exception","reason":"object mapping for [log] tried to parse field [log] as object, but found a concrete value"}}}]}Note If you happen to have a fresh index, you MAY get the opposite behavior, where Line 2 will be indexed first and cause the "log" field type to be "text", in which case mapped objects will fail with an error like Can't get text on a START_OBJECT at 1:49
I presume that the log field should always be text, and that the mapped object should be merged into the top level data structure, as the documentation describes.
edsiper
sestrel
gabe-sorensen
I've read the documentation rearding the nested JSON support you are currently implementing for 0.13 in form of parser decoders: http://fluentbit.io/documentation/0.13/parser/decoder.html
You describe a typical use case in form of a containerized environments with Docker where the application logs data in JSON format which becomes an escaped string.
When using the "Decode_Field json log" there is one problem here, everything written to sysout/syserr is being written to disk by the docker json-file log driver which is being used to accomplish the described use case. But, not every line written to sysout/syserr is a well formatted JSON message. This leads to:
if the content is not a valid json-content the "log" attribute is of type "string"
if the content is a well formed json the "log" attribute is of type object
So why is this a problem? We do all that work because we want to collect all those logs and make em searchable through elasticsearch or something else. But the type of a json attribute has to stay the same for being indexable with elasticsearch. It has to stay either a string or an object.
Are there any plans for a remapping support of the decoded json content? The kubernetes filter for example merges all fields into the parent locument. Here we could do something similiar.