Accept JSON as an input to the collector

[rationalBean]

There is currently no capability to use JSON data as an input - the collector sees the entire JSON string as as single message.

Instead of having to re-parse the JSON'ized message using a Graylog server extractor, provide a capability to add the JSON data as additional fields in the GELF message before sending it to Graylog server.

[bernd]

Can you show us an example JSON file and tell us which program is generating this?

We are trying to minimize the processing on the collector so this is probably something we will not implement except there are good reasons.

[razvanphp]

I think I have the same feature request.

Currently we are using log2gelf to send messages already stored in JSON format from varnish. The varnishncsa daemon can be configured to store one message per line, so we already formatted it as GELF, we just need something to pick them up and send them to graylog reliably (when graylog is down, queue them, resume where it left last time etc)

The problem with current solution is that it uses more CPU than varnish itself... and needs to be manually restarted every time graylog goes down.

I don't see how this feature adds more processing time, since the message is already formatted as it should, it just needs to be somehow defined as json file format and attributes need to be correctly detected.

Thank you!

[joschi]

Since Graylog 1.2.0, we ship a JSON Extractor which can be used to "expand" a JSON payload received as plaintext into a proper GELF message. You can read in the varnish log file with the Graylog Collector and use a JSON Extractor on the GELF TCP/UDP input.

[razvanphp]

OK, thank you, I will try this solution, but I still think this should be the collector's job, this way we would avoid unnecessary processing power on the server.

[joschi]

We designed the Graylog Collector exactly the other way round. The goal was to do as little preprocessing on the client machines as possible and move that complexity into the Graylog server nodes.

[razvanphp]

I tried the solution, it works great, thank you!

[et304383]

I'm struggling with this like crazy. I just want to send in some flat json to Graylog and have it fill in GELF info by extractor. I can almost send data in but it's like its discarding it as I get no recent message errors from the Extract page when clicking load message.

The documentation really is lacking here. A full fledged example of how to send JSON directly into Graylog would be extremely useful.

I'm trying to send AWS CloudTrail logs directly from S3 into Graylog (via Lambda). No, I'm not using the plugin which assumes a horribly cost inefficient model of CloudTrail -> SNS -> SQS <- Graylog (via pull).

[joschi]

@et304383 We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!