search

tsaikd / gogstash

Logstash like, written in golang
MIT License
644 stars 106 forks source link

Input - UDP packet mode #146

Closed helgeolav closed 3 years ago

helgeolav commented 3 years ago

I am missing a way to handle packet based messages in UDP. (Think syslog etc.) Input socket only handles line delimited inputs, and this does not work well with our input. I have been looking through the code and I think the best way to solve this is to create a new UDP-only input type. (The alternative I looked into is to change socket to be more packet-like for UDP connections.)

In addition I could also add support for syslog parsing as a codec, filter or both using go-syslog.

Will this be an acceptable solution?

tengattack commented 3 years ago

try: https://github.com/tsaikd/gogstash/tree/master/input/socket

It can use UDP connection.

helgeolav commented 3 years ago

The problem with socket is in the first line of the documentation: Input event message should end with new line (\n).

If UDP packets arrive and does not have \n in them gogstash will just keep them and fill up the memory. But it does not produce any output. Syslog messages does not end with \n.

The reason for me suggesting to write a new input is that socket is stream driven and I will need to make several changes to make it packet-oriented. My plan was to use net.PacketConn interface so I also can collect the source IP for each packet and add it as a field.

helgeolav commented 3 years ago

Implemented in #152.