search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.35k stars 1.06k forks source link

Support multiple JSON paths in "JSON path from HTTP API input" #650

Open ghost opened 10 years ago

ghost commented 10 years ago

The current implementation of the JSON path from HTTP API input is very limiting, we can get only one value from the JSON response with one JSON path. It would be desirable to be able to specify multiple JSON path->message field mappings.

mikenabhan commented 5 years ago

Is there any update on this? Similarly, even if a single path specifies multiple values, the messages do not come through. There doesn't seem to be a good way to ingest json data from an HTTP API.

ghost commented 5 years ago

Nowadays it's possible to implement custom pipeline functions directly in Java if needed – something like this should be fairly easy to write. It also fits the pipeline approach better; as Graylog is now, I fail to see why this should be implemented at the input level. 4 years ago it was a different story.

mikenabhan commented 5 years ago

Yes, I agree the pipeline function is the way to parse this data, but the jsonpath function in graylog will only allow for a single datapoint. It can't ingest the entire json stream, and fails to work if a path with multiple values is specified. That is the issue I am trying to solve.

http://jsonpath.herokuapp.com/?path=$.store.book[0]

This path, for example, specifies everything associated with the first book, but this sort of syntax does not appear to work in graylog.

Perhaps I am misunderstanding, or this was not the correct place to post about this.

ghost commented 5 years ago

On Fri, Dec 07, 2018 at 11:36:56AM -0800, Mike Nabhan wrote:

Yes, I agree the pipeline function is the way to parse this data, but the jsonpath function in graylog will only allow for a single datapoint. It can't ingest the entire json stream, and fails to work if a path with multiple values is specified. That is the issue I am trying to solve.

Exactly, that is why I was referring to 'custom pipeline functions' -- it's simple to implement a custom function that does what you need in Java and invoke it from the Graylog DSL. You'd write a Graylog plugin [1].

I hit a similar issue with a function(ality) limitation in the DSL a while back (#3605) and judging from the reaction there the developers won't modify 'simple' built-ins to cover all imaginable use cases.

If you need to solve your problem now, writing such a custom function is probably the most efficient way to go about it.

Perhaps I am misunderstanding, or this was not the correct place to post about this.

Since this issue here deals with an input, it is probably better to open a separate ticket as a feature request concerning the pipeline function (may get dealt with and/or implemented faster).

Inputs are different modules of the Graylog system and Torch are working hard to move get rid of processing functionality in inputs/extractors and fully migrate to the pipeline-based system, which is why I would keep our two issues separate.

[1] https://www.graylog.org/post/writing-your-own-graylog-processing-pipeline-functions