JSON parsing should not fail on missing fields

[lennartkoopmann]

Problem description

When parsing a JSON message, the pipeline rule fails if a field does not exist, but it defined in select_jsonpath.

For example, this will fail:

let json_result = parse_json(to_string($message.message));
let json_fields = select_jsonpath(json_result,
            { source: "$.source",
              some_field: "$.i_dont_exist"
            });

The error message will be:

For rule 'f5_test_rule': In call to function 'select_jsonpath' at 8:22 an exception was thrown: null

I suggest ignoring missing fields by default and introducing a new optional boolean parameter to toggle this behaviour.

Environment

• Graylog Version: 2.3.1
• Pipeline Processor plugin version: 2.3.1

[joschi]

@lennartkoopmann I'm unable to reproduce this issue with Graylog 2.3.1 and 2.4.0-SNAPSHOT.

The test case I've been using is directly derived from your example:

diff --git a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
index e036970..75c4b7c 100644
--- a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
+++ b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
@@ -7,4 +7,7 @@ then
               author_last: "$['store']['book'][-1:]['author']"
             });
   set_fields(new_fields);
+
+  // Don't fail on missing field
+  let missing_field = select_jsonpath(x, { some_field: "$.i_dont_exist" });
 end
\ No newline at end of file

It looks like json_result was null when you tried running the JSON path extraction on it.

The parse_json() function may return null if the input wasn't valid JSON: https://github.com/Graylog2/graylog-plugin-pipeline-processor/blob/9fe21fae59eeeda56ac549871683c31ded3d4e73/plugin/src/main/java/org/graylog/plugins/pipelineprocessor/functions/json/JsonParse.java#L51-L56

This all being said, I think the parse_json() function shouldn't return null and the json_path() function should be able to handle null as an input.