search

Azure / azure-diagnostics-tools

Plugins and tools for collecting, processing, managing, and visualizing diagnostics data and configuration
98 stars 92 forks source link

Graylog GELF warning "is missing mandatory "host" field" #140

Closed hassanbabaie closed 6 years ago

hassanbabaie commented 6 years ago

I have logstash with the azureblob plugin and it is forwarding NSG Flow logs to Graylog and I am seeing the following message fill up my graylog server.log

2018-02-01T12:20:55.232-05:00 WARN [GelfCodec] GELF message <42bd8ec5-0774-11e8-ace9-000d3a18d769> (received from <172.18.4.17:50998>) is missing mandatory "host" field.

It appears to be related to how graylog 2.3.1 and up is now set to alert on a missing mandatory field. What I can;t tell is this is something that needs to be resolved in the plugin?

Internal logs filling after update to 2.3.1 https://community.graylog.org/t/internal-logs-filling-after-update-to-2-3-1/2609/4

Any thoughts welcome.

Hass

brahmnes commented 6 years ago

@hassanbabaie I am not familiar with Graylog enough to understand this issue. If it's just a missing field, it's possible to write a filter to add this field. If the NSG log just doesn't have this information, then I don't know how you can solve this other than faking the field, or use an empty string. So I don't see this being something to be solved in the azureblob plugin.

hassanbabaie commented 6 years ago

Thanks @brahmnes I did trying forcing a host field to being added to every record but it didn't seem to help.

I'll post this to the Graylog Extended Log Format (GELF) folks and see what they come back within and will provide an update on this ticket. It doesn;t seem to be causing an impact just filling the server event logs.

Thanks

brahmnes commented 6 years ago

Have you tried using a logstash filter?

hassanbabaie commented 6 years ago

Thans @brahmnes no not tried using that but I will take a look now and if it may help. However I've also asked for more information in the Graylog forum. Ticket linked below:

https://community.graylog.org/t/graylog-gelf-warning-is-missing-mandatory-host-field/4033

hassanbabaie commented 6 years ago

Just found this which might do the trick:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-gelf.html

I assume I would only need to change the output section of my conf

hassanbabaie commented 6 years ago

I added a static field for 'host' within the logstash.conf and this has resolved the issue