Closed nicklit closed 1 month ago
How about specifying auto_increment_key
?
<source>
@type sample
tag log
sample {"message": "{\"field\":\"value\"}"}
auto_increment_key key
</source>
<filter log>
@type parser
key_name message
reserve_data true
remove_key_name_field true
hash_value_field data
<parse>
@type json
</parse>
</filter>
<match log>
@type stdout
</match>
It seems that sample input is reused and it causes such a error.
TODO: maybe in_sample implementation should be improved.
@nicklit Thanks for your report!
I have confirmed that the pair of in_sample
and filter_parser
causes this issue.
I'd like to confirm one thing.
Are you having similar problems with other input plugins?
(Is in_sample
plugin in this issue just an example?)
We can consider this a problem on filter_parser
's part.
We can think that the responsibility for changes to Fluentd's events should lie with the plugin that makes the changes.
Actually, filter_record_transformer
copies the incoming events by itself.
I'm not seeing the problem with other input plugins, just sample. My other input plugins (in_syslog) are typically just a one and done thing though, where it seems like sample is using the same modified input rather than new inputs on each run like I'd expect? It was frustrating/confusing when trying to debug another issue I was having with in_syslog to see the constant error stream (I learned the hard way it was limited to 2048 characters by default). Good idea on the auto_increment_key
workaround @kenhys, I do feel like it's still a bug though.
Describe the bug
When trying to parse a json log in message to a new field, if you try to remove the message field it works the first time, then fails every time after it runs
To Reproduce
copy the config file below and restart td-agent
Expected behavior
no error. message field gone and data field only existing every time sample runs 2024-07-31 16:12:44.030559702 -0400 log: {"data":{"field":"value"}}
Your Environment
Your Configuration
Your Error Log
Additional context
I didn't see anywhere in the fluentd docs where message has to exist. The other ticket I found relating to this error looked specific to an elasticsearch output plugin