Duplicating a message with create_message() loses all fields

[sjahl]

Problem description

I have a use case, where I need to remove a sensitive field from a log message, while still maintaining the content of the original message (this is because there is certain information that only privileged users at my company are allowed to see). In a message processing pipeline, I'd like to be able to duplicate a message and all its fields, routing the original copy to one stream, and the filtered copy to another, unprivileged stream.

However, when using create_message(), all fields from the original message are lost, making field removal on the new message impossible.

Steps to reproduce the problem

1. Enable the Random HTTP Message Generator input
2. Configure two destination streams, one for unfiltered messages, another for filtered.
3. Set up a pipeline processing rule that looks like:

rule "filter user ids"
when
  has_field("user_id")
then
  route_to_stream(id: "<stream id 1>");
  let x = create_message();
  remove_field("user_id", x);
  route_to_stream(id: "<stream id 2>", message: x);
end

My expectation is that the message created and stored in the x variable has all the fields that the original message had, so I can remove_field() on the user_id field. But, the new message only has a message, source, and timestamp field. Perhaps another function (i.e. duplicate_message()) is desireable.

The ultimate goal here is to enable field level access control -- so another way to do this that doesn't involve duplicating messages in the ES index would be great!

Environment

• Graylog Version: 2.1.2+50e449a
• Pipeline Processor plugin version: Unsure of how to determine this
• Elasticsearch Version: 2.4.2
• MongoDB Version: 2.6.10
• Operating System: Ubuntu 16.04.1 LTS
• Browser version:

[g-londhe]

Hi,

Can anybody guide me to where do I need to put those rules, like in which file?

I want to modify the create_message method and modify the input format of Graylog.