Field name suffix "_converted" causes problems

[patrickmann]

The pipeline rules parser is confused by appending the suffix _converted to a field name.

Expected Behavior

I can use all alphanumeric characters for field names, without any problems.

Current Behavior

According to Graylog2/graylog2-server#6391 field names including a dash need to be single-quoted. This is already surprising and not documented anywhere that I am aware of.

Apparently there are other things that confuse the pipeline processor when parsing field names (see this community forum post).

Steps to Reproduce (for bugs)

1. Create a pipeline for all messages
2. Create a rule and add the following statement, where numfield is the name of an existing numeric message field: set_field("numfield_converted", "x"); The problem does not exist for non-numeric fields.
3. Observe that no more messages appear in search
4. Observe that nothing appears in the processing and index failures stream
5. Wrap the offending field name in single backticks: set_field("`numfield_converted`", "x");
6. Observe that messages re-appear in search, but still do not contain the new field.

Context

This is extremely confusing and results in hours wasted debugging

Your Environment

• Graylog Version: 4.3
• Java Version:
• Elasticsearch Version:
• MongoDB Version:
• Operating System:
• Browser version:

[bernd]

@patrickmann Could this be a red herring and the actual issue is that the set_field call sets a value with an incompatible type, so the message cannot be indexed anymore? I would be surprised if _converted has a special meaning in the pipeline language. :smile:

[patrickmann]

I verified that this only affects version 4.3 (and presumably prior versions). Version 4.4 does not exhibit the behavior.

[patrickmann]

Turns out the behavior is an artefact of ES rather unintuitive dynamic typing system. This has nothing to do with handling of variable names in the pipeline processor. The problematic field must have been first assigned type A; subsequently, attempting to assign a value of type B results in a processing error like this:

This is also explained nicely here.