Messages validated against stream but not routed into it

[michelealbrigo]

Expected Behavior

I configured a stream which requires message to have both

• facility: local7
• gl2_source_input: input_id (my setup is single-node, single server)

The same stream has "remove from all messages stream" flagged. A message testing ok against said stream should be routed into it.

Current Behavior

The message isn't routed into the specific stream, but it's left in the "all messages" stream. I've tried many different combinations of rules (e.g. matching source with a regex, overwriting source with a lookup table, etc.), but nothing prevents those messages from being routed into the all messages stream, which I use as a "catch all" for all the un-dealt situations.

Steps to Reproduce (for bugs)

1. configure something that sends syslog on UDP-514
2. have that sender not writing a source (i.e.: source = IP)
3. try to route that message into something else than All messages

Context

Since I have some Cisco IOS 12 switches where I can't set the log origin as a hostname, I am trying to route their syslog messages into the same stream and index of all our other switches (IOS 15 and above). Regular switches (15+) log on syslog tcp-514, with hostname as source. These switches log on syslog udp-514, with IP as source.

Your Environment

• Graylog Version: 2.3.1+9f2c6ef
• Elasticsearch Version: 5.5.2
• MongoDB Version: 1:2.4.10-5+deb8u1
• Operating System: Debian 8.9
• Browser version: Opera 47.0.2631.71 Mac

[joschi]

@michelealbrigo Stream rules are only applied to new incoming messages. Old messages which have already been ingested before the stream or the stream rules have been created, will not be shown in the stream.

This being said, are you using any extractors or pipeline rules which modify the facility or gl2_source_input fields?

[michelealbrigo]

The stream rule was older than the message, which was 5/10 minutes old (to be sure that elasticsearch indexed it). The node itself is oversized for the load, so I do not expect processing delays.

I do not use any pipeline (I still need to understand them properly) and my extractors do not modify any field at the moment.

-- Michele Albrigo

Il 4 set 2017, 20:06 +0200, Jochen Schalanda notifications@github.com, ha scritto:

@michelealbrigo Stream rules are only applied to new incoming messages. Old messages which have already been ingested before the stream or the stream rules have been created, will not be shown in the stream. This being said, are you using any extractors or pipeline rules which modify the facility or gl2_source_input fields? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[joschi]

@michelealbrigo What happens if you send a handcrafted message matching the stream rule to the Syslog UDP input?

Example:

# Send "Informational" message to facility LOCAL7
# echo "<190>1 $(date --iso-8601=seconds -u) example.com - - - - test message" | nc -u -w0 graylog.example.com 514

[michelealbrigo]

Here I am with some further tests:

• sending a syslog message from another server (i.e. the command you gave me), to the UDP514 input BUT with a properly formatted source, the message gets routed into the correct stream (the one for all the Cisco IOS 15 stuff)
• I built an extractor to populate a "sourceDNS" field with the result of a table lookup, and if I match messages against the sourceDNS field they get routed into the correct stream
• if I fake a message with one of the "known switches" as the source, using the IP in the command you gave me, the lookup works and the message gets routed into the correct stream, being matched against the sourceDNS field
• if I revert to facility = local7 and source = 10.254.x.y as stream rules, both known and unknown switches get routed into the stream
• if I flag the SwitchIOS12 stream to remove messages from the "All messages" stream, I start getting different behaviors, from messages still being routed into the two streams (10 minutes after the stream configuration change, on a system with no backlog), mostly if they are from known switches, to my fake and unknown source being correctly routed in the SwitchIOS12 stream only, to still-unmanaged switches being routed just into the All messages stream, even if the message would have matched the stream rules

[joschi]

@michelealbrigo Please verify that the system clocks on all relevant systems are in sync (e. g. using NTP) and that the timezones used on the network appliances are correct.

Also make sure that the syslog message do indeed include the correct timezone in their timestamps. For this purpose you can activate storing the complete syslog message in the "full_message" field in the configuration of the Syslog UDP input.

[michelealbrigo]

All those switches are synced with our NTP servers. The timezone appears correct, and so does the on-board clock. I've turned on the full_message, but while other UDP514 sources are already showing the field, the IOS 12 switches still show no trace of the full_message field, even 15m after changing the setting.

I get some messages with a malformed source, but I am not sure whether these two things are related.

[joschi]

@michelealbrigo The timestamp of the message says 10:06:59.238 (UTC) in "timestamp" while the raw syslog message in "full_message" says 12:10:14 (UTC).

I'd say that's neither in sync, nor using the correct timezone (also see #3091, #3853).

[michelealbrigo]

Double checking everything, I realized the Graylog server itself wasn't running on NTP-synced time, so I fixed that and went out to lunch. Now messages (at least most of them, in good percentage) are routed into the correct stream, with the "unassuming" stream rules (i.e. facility and source IP), which is the expected behaviour for our setup, a sort of catch-all for our legacy switches.

I've tried fiddling with the timezone and clock settings of our most talkative switch, but this probably wasn't critical to solve our problem, activating NTP on the Graylog server was.

I have two residual issues:

• I receive some messages on the "All messages" stream which have a bogus source, and no immediate way to identify it, no matter the full_message field being turned on
• with the correct time on Graylog server but without the correct timezone setting on the switches, the logs are processed according to rules (i.e. SwitchIOS12 stream, and only that), but I get no full_message field: I will align configurations on all our switches, so this is not important to us, but if this is unexpected behaviour to you, I can provide any further data you need to debug it, and check for solutions (top message is from an untouched switch, bottom one is the one where I did my timezone configuration tests)

[joschi]

I receive some messages on the "All messages" stream which have a bogus source, and no immediate way to identify it, no matter the full_message field being turned on

You can use the gl2_remote_ip field to identify these sources. The field is "hidden" by default, so you'd need to use a Copy Input extractor to copy its contents into another field (e. g. named source_ip_address).

Thanks for your feedback! I'll close this issue now as it's not a bug in Graylog.

If you have more questions around these topics, please create a topic in our community discussion forums.