search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.35k stars 1.06k forks source link

More information in "Unknown beats protocol X"-exception #4763

Open rogermartensson opened 6 years ago

rogermartensson commented 6 years ago

Expected Behavior

I have some beats with "Unknown" protocol version that Graylog 2.4.3 does not understand. I need some more information of the beat sending agent displayed and reduced output in the logs. All this if possible.

Current Behavior

Current behaviour is like this: 2018-04-26T11:14:42.793+02:00 ERROR [NettyTransport] Error in Input [Beats/5ad61f82a3f2bb438c2d992a] (channel [id: 0x6a2af37f, /remote:port :> /graylognode:port]) java.lang.Exception: Unknown beats protocol version: 3

I have the input behind a load balancer and the address in the error message is showing the load balancers ip. ## Possible Solution If the information can be found in the received message about who actually is sending this unknown beat protocol it would be nice to have it displayed. It's not that very helpful to know which load balancer is relaying the message. I hope this is possible to do otherwise I'll have to live with it. It would also be nice to see if the java stack dump about "unknown beat protocol" could be masked out to only show an error. This to decrease number of rows written to log. ## Steps to Reproduce (for bugs) no steps given ## Context ## Your Environment Graylog with different beat versions sending data from clients. Beats is sent to a load balanced address and relayed to graylog nodes. * Graylog Version: 2.4.3 * Elasticsearch Version: 5.6.8 * MongoDB Version: 2.6.10 * Operating System: Ubuntu 16.04 * Browser version: not needed
joschi commented 6 years ago

If the information can be found in the received message about who actually is sending this unknown beat protocol it would be nice to have it displayed.

Unfortunately TCP/IP doesn't include any information about the origin host and doesn't support adding this information to TCP packets.

What we could do, though, is logging a hexdump of the received Beats packet (at least the first few bytes) on DEBUG level, so that interested parties can investigate further.

rogermartensson commented 6 years ago

If the data could be parsed (you do get the beat procol version) then maybe there is some destination in the parseable data that contains the source of the message. But if that isn't possible then if the hexdump(ascii-dump?) contains any information that could help finding the source of the message then it would be nice to have in the logs. If not then there is no reason for a hex-dump.

But then.. This is only when hiding behind a load balancer but I guess I'm not the only one having such a configuration.

It also would be nice to replace the stack trace with a simple yet informative log entry.

joschi commented 6 years ago

If the data could be parsed (you do get the beat procol version) then maybe there is some destination in the parseable data that contains the source of the message.

The problem is that there are currently only 2 versions of the Beats (formerly Lumberjack) protocol, which means that protocol version "3" is invalid in the first place.