search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Pipeline "route_to_stream()" function does not work with custom index sets #4954

Closed lennartkoopmann closed 4 years ago

lennartkoopmann commented 6 years ago

The route_to_stream() pipeline function does not route the message into the index set of the underlying stream.

Expected Behavior

When routing a message into a stream that has a custom index set configured, I expect the route_to_stream() function to write the message into that index set.

Current Behavior

The message is written to the default index set.

Steps to Reproduce (for bugs)

  1. Create a stream with a new custom index set.
  2. Route a message into that stream, using the route_to_stream() function.
  3. Open the index set and you'll see that it has not stored any messages.
  4. Open the stream. You will not find any messages.
  5. Change the stream setting to use the default stream and you'll find the routed messages because Graylog is now searching in the default index set.

Your Environment

bernd commented 6 years ago

@lennartkoopmann I am unable to reproduce that. Can you please show us your pipeline rule code?

lennartkoopmann commented 6 years ago

I've had another issue very much related to this one. Will try to find time today or tomorrow to write up the exact steps to reproduce. (I think the processor order might be involved.)

Stay tuned.

jalogisch commented 6 years ago

I tried to reproduce this in my Lab and a clean OVA installation and can't reproduce this. Would need to know how this can happen, or what you configured how exactly.

lennartkoopmann commented 6 years ago

so....... :) I tried to reproduce this again and now it just works. Not sure what I did differently the last time.

Thanks!

jrunu commented 5 years ago

I hope its alright that I bump this closed issue. I was seeing exactly the same behaviour described in the Reproducing Steps in the original post. Only that I'm running a graylog 3.0.2+1686930.

While investigating I determined the following behaviour to be consistent:

  1. Setting a different Index Set while Creating stream leads to the expected behaviour
  2. Changing the Index Set after creation shows the unexpected behaviour.
  3. Pausing / Unpausing the Stream after performing the index set change leads to the expected behaviour.

So I would assume that, unlike with editing and saving for example inputs, the trigger to restart/reload/propagade the change is not triggered.

bernd commented 5 years ago

@jrunu Thank you for the updated steps to reproduce this.! :+1: I will reopen the issue.

thll commented 4 years ago

I was finally able to reproduce the issue. It only happens if the route_to_stream() pipeline function routes to the stream by name . If it routes to the stream by ID, everything works as expected. This is a bug in the internal caching of streams used by the function. We will provide a fix for that.

lennartkoopmann commented 4 years ago

That's great news! Thank you.