teragrep / lsh_01

logstash-http-input to syslog bridge
https://teragrep.com
Apache License 2.0
0 stars 3 forks source link

json_batch not supported #75

Closed kortemik closed 1 month ago

kortemik commented 2 months ago

Describe the bug json_batch not supported. current feature supports regex splitting but not json compatible array splitting.

Expected behavior configuration should let decide if regex or json based processing is in use.

How to reproduce

Screenshots

Software version 3.2.1

Desktop (please complete the following information if relevant):

Additional context

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html#plugins-outputs-http-format

[
{"a": 1},
{"b": 2}
]
51-code commented 2 months ago

Config parameters are to be changed: payload.splitEnabled is to be removed. payload.splitType is added. Value is either regex, json_array or none. none is the default. payload.splitRegex is moved to payload.splitType.regex.pattern.

If json_array needs configuration, it's going to be put in payload.splitType.json_array.

51-code commented 2 months ago

Refactored how splitting the messages works in RelpConversion object. It was quite bad before, with the regex splitting just built in the object. I refactored it so that it is a decorator called RegexConversion, which splits the message before using the regular RelpConversion. This is better design altogether and now the json_array splitting can be added similarly.

51-code commented 2 months ago

Refactored Payload objects and created end to end unit tests for the json_array splitType.

51-code commented 2 months ago

There was some difficulties with getting the JSON objects as Strings. The in-built toString() function in Jakarta takes out any whitespace in the input. However, this was assessed to be okay, and the implementation will continue using the toString() function.

51-code commented 1 month ago

Draft PR done, assistance needed for a minor design issue.

51-code commented 1 month ago

Adding new configuration options for this functionality brought up the problem with the current configuration objects. I opened a new issue

about them. The new configuration options are already using this new style which is explained in the issue.