Open drewmiranda-gl opened 2 years ago
The "JSON path from HTTP API" input was designed to store a single primitive value, not a complete JSON payload.
If the value returned by the JSONPath setting is a nested JSON structure, the codec stores a string representation of the Java Map
structure in the result
field of the message. The string representation of a Java map is not valid JSON.
We cannot change the current behavior because a change can break existing setups. We could introduce an input configuration setting to store map payloads as JSON, but I think it would be better to create a new input type that is designed to fetch and process JSON payloads.
The current input was basically designed to fetch download counts from GitHub. (as shown in the docs)
@bernd Thanks for the background info. Are you thinking that the separate input would also support doing more extended JSON processing (above what the current one does)?
If it's mostly the same (with added JSON element targeting), then the configuration option approach for this input seems like an OK backwards-compatible solution to me, since the Java map representation seems like a bug/unintended side-effect, and it would provide a workaround to get a predicable format with this input. Seems easy enough to do from a code-perspective. I know the original intention of the input was different, but this seems to still fit with JSON path functionality.
One of the main benefits is that the json_flatten
pipeline function could then be used to add fields for a specific JSON element (instead of the whole document).
But, I also think that if we have planned more comprehensive JSON processing support, then a separate input would make sense.
But, I also think that if we have planned more comprehensive JSON processing support, then a separate input would make sense.
@danotorrey I don't think we have anything explicitly planned yet. I was thinking about having more flexibility regarding the configuration.
A few examples I had in mind:
result
key. Users might want to configure that.List
, the result
only contains the first element of the list. Similar to the map issue, this might not be what users expect.POST
requests with a payloadBut you are right; we could add an option to enable the "JSON value mode" and adjust the map and list handling depending on the value of that option.
@bernd Good point, there are quite a few limitations with this input. A new input addressing these would be a lot more usable.
Do you have any gut feeling for what to do about this issue? My vote is to consider adjusting the display name for the input to "JSON path value from HTTP API" and add a small clarifying note to the documentation about only supporting primitive values. Then, save the effort of more extensive JSON handling improvements for the new input.
@danotorrey I think you should talk to product and check the priority and next steps.
My vote is to consider adjusting the display name for the input to "JSON path value from HTTP API" and add a small clarifying note to the documentation about only supporting primitive values.
Sounds like a good first step 👍 Could the SecDev team drive this?
Then, save the effort of more extensive JSON handling improvements for the new input.
I cannot say much about the priority for that right now. It sounds like a useful addition though.
@boosty
Sounds like a good first step 👍 Could the SecDev team drive this?
Definitely. PR has been opened and will be backported too: https://github.com/Graylog2/graylog2-server/pull/13439 I will coordinate the docs changes too.
I cannot say much about the priority for that right now. It sounds like a useful addition though.
If it makes sense, we can keep this issue open for future consideration.
When using the input type
JSON path from HTTP API
, you are required to provide a value forJSON path of data to extract
.However, any value placed in this field other than
*
returns invalid json that cannot be parsed.Expected Behavior
Json selected using JSON path returns valid json that can be parsed using json parsing functions via processing pipelines.
Current Behavior
Attempting to parse json returned produces an error:
Possible Solution
Steps to Reproduce (for bugs)
JSON path from HTTP API
inputhttps://randomuser.me/api/
$.results.0
(this can be validated as valid json path via https://jsonpath.com/ )Your Environment
Please let me know if there are any questions or additional information needed.