search

tsaikd / gogstash

Logstash like, written in golang
MIT License
644 stars 106 forks source link

Do you have any plan to add plain codec? #198

Open yjagdale opened 2 years ago

yjagdale commented 2 years ago

Do you have any plan to add plain codec?

helgeolav commented 2 years ago

Hi

Could you tell a bit more about what you are trying to do? What inputs/outputs you want to use a plain codec on?

I assume that with "plain" you mean that a text string should not be handled in the input/output module.

yjagdale commented 2 years ago

so right now every input which is coming from the source is converted to json (timestamp). In my case, I don't want that conversion and input -> filter it should be the same string as the source.

yjagdale commented 2 years ago

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-plain.html is the original codec from elastic.

helgeolav commented 2 years ago

Ok, can you tell me what inputs and outputs you work with?

An event contains several fields. Timestamp and message are the two most used.

An input without a codec will typically place the input data into the field message. If the same input is configured to use the json codec, the input data will be parsed and new fields will be created for each set of data in the json.

A timestamp is always added to the event on input.

The filters work on whatever field you want it to work on.

On the output side it is somewhat similar. The elastic output will always convert the event into a JSON struct, meaning all fields, including the timestamp, will be included. Other outputs can handle this differently, allowing you to either choose a codec (json vs plain text), or what field to push out.

helgeolav commented 1 year ago

The filters grok and gonx can be used instead of a codec to parse lines of text.