search

emsearcy / fluent-plugin-gelf

Buffered fluentd output plugin to GELF (Graylog2)
Apache License 2.0
33 stars 57 forks source link

Unextracted Message Field in Graylog #3

Closed humpalum closed 7 years ago

humpalum commented 9 years ago

Hi, There is a problem with extracting Fields in Graylog. You can see the issue in the following recipe: http://www.fluentd.org/guides/recipes/graylog2

The last picture shows the Message-Field, which looks very messed up: http://www.fluentd.org/images/graylog2-graph.png I tried it at my own system and it looks exactly like in the last picture.

When using the following nc example message it looks all fine: echo '{"version": "1.1","host":"example.org","short_message":"A short message that helps you identify what is going on","full_message":"Backtrace here\n\nmore stuff","level":1,"_user_id":9001,"_some_info":"foo","_some_env_var":"bar"}' | nc -w 1 -u localhost 12200

So my guess is that either out_gelf plugin is bugged or the ruby gem gelf.

regards.

emsearcy commented 9 years ago

Hi! Can you see if the patch at https://github.com/emsearcy/gelf-rb/commit/81656c2f554c1878607210f2b0b2912dd8b9df49 makes a difference? I forget whether I made a pull request for that, I will go double-check.

jonathancrosmer commented 9 years ago

I ran into this same issue. It seems like a bug in out_gelf, or else I don't understand how it works. The fluentd output is double-encoded so that the log message in Graylog appears as some kind of JSON. When outputting the GELF JSON, shouldn't the fields from the log message be included separately, not as one giant "message" field? Also @emsearcy how would the logger levels make a difference? It seems like the plugin is outputting JSON where one of the values is also the JSON input string, instead of extracting values from the JSON input and adding them to the output.

jonathancrosmer commented 9 years ago

After inspecting the source I see that the input must have a short_message field, or else the short_message in the GELF JSON output is set to a dump of the whole JSON input, which was confusing. In my case, I had a simple "tail" input using format none, so I just added message_key short_message on the next line, then Graylog showed the correct value. I suppose the example linked above had the same problem, which is that there was no short_message, so the plugin dumped the whole JSON string as the short_message. That confused me.

emsearcy commented 9 years ago

Oh, yes you're right that has nothing to do with the logger levels.

Yes, GELF documentation states that short_message is a required field and "MUST" be included, so if it is not included I dumped the JSON to populate this field. The purpose was not double-encoding, but to put something useful into short_message.

Are you using in_gelftail? If you notice my included formats in in_gelftail have a named regex capture for short_message (and full_message) which span several other fields, so that the original line is preserved for the Graylog web interface.

Aside from documenting this better, I'd welcome suggestions about how to build a default short_message.

jonathancrosmer commented 9 years ago

I'm totally new to FluentD and Graylog. I haven't used in_gelftail yet, but it sounds useful. At the moment I'm just trying to set up our system to get our existing web application log into Graylog.

Now that I understand it, it seems like this is a warning condition to me. It's friendly to not fail completely, but it is a kind of misconfiguration without an obvious fix. My tendency would be to include the warning (so you know your setup is broken) plus the JSON dump (just to be friendly, so you can see something). Also, I might try to make the warning string somewhat unique, so if someone googles it he'll quickly figure out what's going on.