Open kvesterling opened 7 years ago
joschi
commented
7 years ago @kvesterling Please attach the pipeline rules you're using and the extractor configuration (if you're using any).
Also, please provide one or more specific examples of such messages. What fields do you expect them to include, what's actually included?
kvesterling
commented
7 years ago As I mentioned, if it's configured to be BEFORE the message processors configuration, everything works perfectly. If it's after, I see the counters messages/sec going through the motions, but fields are not set properly.
That being said, there's the rule: --begin-- rule "checkpoint" when from_input("5980ac1d97fe9532d3ca196b") then set_fields(key_value(to_string($message.message), "|", "=", true, true, "", " ", " ")); set_field("timestamp", parse_date(to_string($message.time), "yyyy-MM-dd HH:mm:ss", "en", "America/New_York")); end --end--
The data (which is PROPERLY PARSED) by the rule above so long as the pipeline is configured to be BEFORE the message processors. I'm sorry, I cannot post logs as it would expose internal workings of our organizations. From your response it seems that you think it's ONE rule, but it's not... It's ALL rules, anything having to do with pipelines is disregarded.
This works: (all rules in pipeline work)
-- | -- | -- 1 | Pipeline Processor | active 2 | Message Filter Chain | active 3 | GeoIP Resolver | active
This does not: (no rules in pipeline work)
-- | -- | -- 1 | Message Filter Chain | active 2 | Pipeline Processor | active 3 | GeoIP Resolver | active
kvesterling
commented
7 years ago It's been 2 weeks... What more input could you need?
joschi
commented
7 years ago @kvesterling We haven't had time to reproduce the problem. You'll see updates in this issue when something has been done.
zez3
commented
4 years ago I can see the same behavior in streams but only for source field. When I try to use source in a stream it is simply ignored so I am forced to create a pipeline and route it. For other fields the stream matching works(e.g. mess types field)
My Message Processors Configuration is: 2 | Message Filter Chain | active 3 | Pipeline Processor | active
I am on graylog Version 3.1.4
Or am I wrong understanding the documentation? https://docs.graylog.org/en/latest/pages/pipelines/stream_connections.html#the-all-messages-stream " if you prefer to use the original stream matching functionality (i.e. stream rules), you can configure the Pipeline Processor to run after the Message Filter Chain (in the Message Processors Configuration section of the System -> Configurations page) and connect pipelines to existing streams. This gives you fine-grained control over the extraction, conversion, and enrichment process. " it is desired but does not seem to work for source field But then the next section states: https://docs.graylog.org/en/latest/pages/pipelines/stream_connections.html#the-importance-of-message-processor-ordering "Message Filter Chain is responsible for setting static fields and running extractors defined on inputs, as well as evaluation of stream rules"
What is a static field ? is source one of the static fields? then this contradict the first statement
zez3
commented
4 years ago I set my static fields in inputs so source is not a static field
Graylog v2.3.0+81f8228 + Ubuntu 16.04
Expected Behavior
The rules and such within the pipelines should be processed, but they are not, or perhaps the results merely not assigned to the message.
Current Behavior
If I have the pipeline processor configured as #1 in the list, all is well... Anything else, I see messages get sent to the pipelines, but seemingly nothing is processed. Put the pipeline processor as #1 in the list, and everything works as expected.