search

Graylog2 / graylog-plugin-pipeline-processor

[DEPRECATED] Graylog Pipeline Message Processor Plugins
https://www.graylog.org/
GNU General Public License v3.0
20 stars 15 forks source link

Pipeline Processor plugin warning in Graylog 2.4.0 #236

Closed colin-stubbs closed 6 years ago

colin-stubbs commented 6 years ago

Problem description

Pipeline Processing does not work at all.

As below; almost all other plugins have been removed. The only error logged relates to the pipeline processs itself.

All pipelines, rules and stream/pipeline connections etc had been deleted to isolate the issue. graylog server restarted clean with ZERO pipeline configuration in GUI or in mongoDB.

Creating a simple rule as below,

rule "test" when has_field("message") then set_field("test","1234"); end

will result in the following error message,

Logs error "WARN [DeadEventLoggingListener] Received unhandled event of type from event bus <AsyncEventBus{graylog-eventbus}>"

Deleting the same rule does not result in a log message.

Create a test pipeline results in the same log message.

Connecting the test pipeline to the "All Messages" stream results in the same log message.

Adding the test rule to the test pipeline results in the same log message.

Using the pipeline simulator with dummy message from "All Messages" stream shows that test pipeline rule should work, e.g.

Simulation results These are the results of processing the loaded message. Processing took 660 µs.

Changes in original message de877dc1-f162-11e7-bc6d-000c29233450 Added fields test 1234

2018-01-05T01:09:53.326+10:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Cb-Protection-Syslog, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992. 2018-01-05T01:09:53.329+10:00 INFO [InputStateListener] Input [Syslog UDP/5a46d2be1d67a30519e24d88] is now RUNNING 2018-01-05T01:09:53.357+10:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=Global-GELF-TCP-12201, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=null} should be 1048576 but is 212992. 2018-01-05T01:09:53.359+10:00 INFO [InputStateListener] Input [GELF TCP/5a1cc21b1d67a3df2de91cfb] is now RUNNING 2018-01-05T01:10:47.418+10:00 WARN [DeadEventLoggingListener] Received unhandled event of type from event bus <AsyncEventBus{graylog-eventbus}> ^C [root@c01 plugin]# ls -l total 11928 -rw-r--r--. 1 root root 6615813 Dec 22 22:29 graylog-plugin-map-widget-2.4.0.jar -rw-r--r--. 1 root root 5592446 Dec 22 22:29 graylog-plugin-pipeline-processor-2.4.0.jar [root@c01 plugin]# rpm -V graylog-server SM5...GT. c /etc/graylog/server/server.conf S.5....T. c /etc/sysconfig/graylog-server missing /usr/share/graylog-server/plugin/graylog-plugin-aws-2.4.0.jar missing /usr/share/graylog-server/plugin/graylog-plugin-beats-2.4.0.jar missing /usr/share/graylog-server/plugin/graylog-plugin-cef-2.4.0.jar missing /usr/share/graylog-server/plugin/graylog-plugin-collector-2.4.0.jar missing /usr/share/graylog-server/plugin/graylog-plugin-enterprise-integration-2.4.0.jar missing /usr/share/graylog-server/plugin/graylog-plugin-netflow-2.4.0.jar missing /usr/share/graylog-server/plugin/graylog-plugin-threatintel-2.4.0.jar [root@c01 plugin]# cat /etc/sysconfig/graylog-server

WARNING: This file is managed by Salt Stack

GRAYLOG_SERVER_ARGS="" GRAYLOG_COMMAND_WRAPPER="" JAVA="/usr/bin/java" GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts"

EOF

[root@c01 plugin]# cat /etc/sysconfig/graylog-server graylog-plugin-^C [root@c01 plugin]# rpm -qa | grep -i -e elasticsearch -e mongo -e graylog mongodb-org-server-3.4.10-1.el7.x86_64 mongodb-org-shell-3.4.10-1.el7.x86_64 mongodb-org-tools-3.4.10-1.el7.x86_64 graylog-server-2.4.0-9.noarch python-pymongo-2.5.2-4.el7.x86_64 elasticsearch-5.6.5-1.noarch mongodb-org-mongos-3.4.10-1.el7.x86_64 mongodb-org-3.4.10-1.el7.x86_64 [root@c01 plugin]#

Steps to reproduce the problem

  1. Install Graylog Server 2.4.0 on CentOS 7/Enterprise Linux 7

Environment

rphlwnk commented 6 years ago

Same issue on creating new pipelins - pipelines from Graylog 2.3 work fine.

Graylog Version: 2.4.0 Pipeline Processor plugin version: 2.4.0 Elasticsearch Version: 5.6.5 MongoDB Version: 3.6.0 Operating System: CentOS 7.4

colin-stubbs commented 6 years ago

Actually, that's the case for the 2.3 to 2.4 systems I upgraded too... the pipeline processing configured as part of 2.3.x or earlier is still working in them now they're on 2.4.x.

I'm not sure about what happens if I try to make changes there - I'm loathe to risk breaking them.

kroepke commented 6 years ago

tl;dr: Everything is fine.

This message is the result of an unfortunate oversight after a refactoring which changed how updated rules, pipelines and pipeline stream connections are handled.

The new state used to be posted on an event bus in the server(s) and then processed asynchronously by some other component. The posting still happens, but nothing is listening anymore because the pipeline state is accessed differently now, thus the event bus complains that some event is posted, but no one read it, because that typically indicates a bug or failure somewhere.

We'll remove the warning in 2.4.1 and 3.0.

Thanks for reporting this!

kroepke commented 6 years ago

@colin-stubbs Aside from the comment I just left, does processing work or not work for you at all?

The first sentence implies that it doesn't, whereas the rest of the post makes it look to me as it does. We've checked with 2.4.0 internally and pipeline processing works just fine here.

colin-stubbs commented 6 years ago

@kroepke it definitely wasn't adding the test field.... but after changing a lot of things including reinstalling the graylog-server package, nuking all streams data caches, lookup tables etc related to the Threat Intel plugin that I was initially trying to use it's come good again.

I'm 100% certain it wasn't working prior to that... pretty sure there's a larger issue there somewhere still.

jalogisch commented 6 years ago

Unfortunately the issue is not reproducible in the latest release. Please re-open this issue if it still occurs, ideally with a set of reproducible steps.

Thanks!

bernd commented 6 years ago

@jalogisch I will re-open this because there is still an open task for 3.0 and 2.4.1. (https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/236#issuecomment-355512366)

I will also change the title for this.