Modify map in pipelines

[fcoelho]

Expected Behavior

I'd like to have a way to modify maps in pipelines. Particularly, I'm looking for ways to do one of:

1. remove a key from a map
2. assign a value to a map key

I'm looking for something that would allow me to do something equivalent to:

// assign/replace
map["timestamp"] = parse_date(map["timestamp"]);
// remove
remove_key(map, "timestamp");

Current Behavior

There's no way to modify map variables after they're created (that I could find in the documentation)

Context

I'm trying to convert a timestamp field present in some log groups messages to an actual date object in a pipeline. See https://community.graylog.org/t/removing-key-from-pipeline-map/12037

Your Environment

• Graylog Version: 3.1.0
• Elasticsearch Version: AWS 6.8
• MongoDB Version: 4.2.0

[jpuskar]

Just a bump (apologies). I'm guessing that it's a documentation issue (or that I just missed it).

[ckristo]

Would love to see this feature - seems missing according to rule function docs.

For others looking to avoid name clashes with predefined fields (like timestamp): set_fields has parameters which allow to set a prefix and postfix

[ckristo]

Had a quick look at the code, grammar for RuleLang resides in graylog2-server/blob/master/graylog2-server/src/main/antlr4/org/graylog/plugins/pipelineprocessor/parser/RuleLang.g4, function implementations in graylog2-server/src/main/java/org/graylog/plugins/pipelineprocessor/functions.

Looking at the grammar, it seems not very easy to add support for them[k] syntax to access a map element. Therefore, I would suggest to introduce a function for that like map_get(k, [default_val]). To be consistent, I would suggest map_remove(k) for deleting a value from the map.

If there is interest in having these, I would try to come up with an implementation.

EDIT: seems I missed that there is indeed index access for arrays foreseen by the grammar (here), so m[k] syntax to access a map element might not be that much of a hassle to add? Implementation of expression is here: graylog2-server/src/main/java/org/graylog/plugins/pipelineprocessor/ast/expressions/IndexedAccessExpression.java

[patrickmann]

@ckristo Just curious, what use cases do you have in mind for this? Looking at the pipeline functions we don't do a lot with maps:

• multi-value returns from table lookup
• jsonpath return values
• to_map for subsequent set_fields

[ckristo]

Hey. Pretty much what @fcoelho wanted to achieve in the original issue - I parsed a JSON file and had a name clash with one of the predefined fields (also with timestamp). I could work around that by renaming the field at the generating source (nginx), but I guess there are log sources out there where this is not possible. The alternative would be to apply a prefix to all fields, but this seemed odd to me (for solving a name clash of a single field).

EDIT: reading the original issue more carefully, my issue was a bit different - converting a string to a date could be done directly on $message using set_field()

Nevertheless, it seems general map manipulation functions could come in handy for all the functions you listed :-)