Closed agiilit-admin closed 11 months ago
The anomaly was first observed with AMQP, but it seems to be reproducible with the tail input plugin.
This is not an issue with any input plugin, rather with the JSON v2 parser itself. The first version of telegraf I was able to reproduce with was v1.21.0, which introduce the object field/tags in #9449.
Adding some debugging it appears that the parser is currently adding the kwh_export/o2 value twice. First, correctly as 0 and again as the kwh_import/o1 568.5 value. The parent node index that is used is somehow wrong. That results in the wrong rename getting used and the o2 value getting updated again and no o1 value.
I believe this is all related to the indexing and lookups that happen, somewhere long the way they get off because of the previous values. Specifically, the call to p.existsInpathResults(parentIndex)
looks up the wrong index and returns o2 value. Those index values come from combineObject() here
Related: there have been incidents where simultaneously with errors in fields also tag values exctracted from the JSON have ended up with wrong tag names, but I am unable to repeat the behavior at the moment. The config involved both object field and object tag. Btw, there were two name objects in the JSON document, one at the top level and one as tags.name
.
That problem (with tags) went away when replacing
[[inputs.tail.json_v2.object]]
path = "@this"
with
[[inputs.tail.json_v2.object]]
path = "{fields,tags}"
If an extra "name":"foo"
is added to each JSON document:
{"fields":{"freq":50.02000045776367,"kwh_export":0,"kwh_import":568.5},"name":"foo"}
{"fields":{"freq":50.0099983215332,"kwh_export":0,"kwh_import":568.5},"name":"foo"}
{"fields":{"freq":50,"kwh_export":0,"kwh_import":568.5},"name":"foo"}
and included with json_v2.object path:
[[inputs.tail.json_v2.object]]
path = "{fields,name}"
it ends up as kwh_import/o1 value in last line of output:
tail,host=ai-2,path=test-input.txt o3=50.02000045776367,o2=0,o1=568.5 1692690843010670721
tail,host=ai-2,path=test-input.txt o3=50.0099983215332,o2=0,o1=568.5 1692690843010756312
tail,host=ai-2,path=test-input.txt o3=50,o2=0,o1="foo" 1692690843010788352
The problem here seems to be that parse results accumulate in p.subPathResults
over multiple calls of parseCriticalPath()
. When length of JSON elements vary a lot (as in the provided examples), probability of erroneous index match increases. Sooner or later existsInpathResults()
starts misbehaving, and produces the results above.
Setting p.subPathResults = nil
in parseCriticalPath()
makes this particular issue go away. Having no experience in telegraf internals (or go, for that matter), I have no suggestions on the correct way to fix it.
Could not replicate above behaviour with json_v2 tests with multiple inputs. Have seen it on amqp_consumer
, though.
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.27.3, Debian 12
Docker
No response
Steps to reproduce
./test-input.txt
as follows:telegraf --debug --config test-config.conf
./test-output.txt
Expected behavior
Each line of output should contain fields o1, o2, o3:
Actual behavior
Last line of output is missing field
o1
:Additional info
Number of decimals in freq field seems to have significance. If input is changed to:
The output is as expected.
Also, there need to be multiple lines before "freq":50 for the anomaly to occur.
Another workaround is to increase decimals for the last freq:
..producing:
The anomaly was first observed with AMQP, but it seems to be reproducible with the tail input plugin.