loki.source.syslog "syslog" errors for syslog UDP unless it has a \n

[rarrr]

What's wrong?

When using UDP on the syslog source Alloy reports: ts=2024-04-12T02:15:06.664708206Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"

edit: the RFC is 5424 A newline is not part of the RFC 5425 spec

Steps to reproduce

Using pfsense, setup syslog to forwarding to Alloy.

If I first forward the logs to a python proxy and add a \n to the end of the UDP, Alloy will accept them.

System information

Ubuntu x86

Software version

Grafana Alloy v1

Configuration

logging {
  level  = "debug"
  format = "logfmt"
}

discovery.relabel "syslog" {
        targets = []

        rule {
                source_labels = ["__syslog_message_hostname"]
                target_label  = "host"
        }
        rule {
                source_labels = ["__syslog_message_app_name"]
                target_label  = "app_name"

        }
}
loki.source.syslog "syslog" {
        listener {
                address               = "0.0.0.0:514"
                protocol              = "udp"
                label_structured_data = true
                labels                = {
                        job = "syslog",
                }
                max_message_length = 0
                use_rfc5424_message = true
        }
        forward_to    = [loki.write.default.receiver]
        relabel_rules = discovery.relabel.syslog.rules
}

loki.write "default" {
        endpoint {
                url = "http://loki:3100/loki/api/v1/push"
        }
        external_labels = {}
}

Logs

ts=2024-04-12T02:15:06.32156681Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.322551528Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.323337812Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.324249975Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.325043217Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.325999295Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.329874883Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.331205845Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.332674301Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.332866181Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333002003Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333107209Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333201586Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333304452Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333381038Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333454747Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333525997Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.333617728Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.654446025Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.654911938Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.655135759Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.655405233Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.655623808Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.657406316Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.657682832Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.657915353Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.658732696Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.65957115Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.660404272Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.661616538Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.662477879Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.663286792Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.663966021Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.664238645Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.664459857Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"
ts=2024-04-12T02:15:06.664708206Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog err="unexpected EOF"

[tpaschalis]

Thanks for opening an issue and doing some investigative work here!

I'm not super familiar with syslog, so any help here is appreciated. This is similar to the issue raised in #451; namely, our Loki components work with a line-oriented logic.

The unexpected EOF error seems to come from the syslog-go package that we're using to parse streas; I'm not sure if this newline logic here is inherited there or it's something that we include as part of our code, I'll have to check.

But my question is, is there some way for the syslog stream to denote the end of messages so that it can be used instead of newlines?

[rarrr]

I should have linked this RFC, Transmission of Syslog Messages over UDP https://www.rfc-editor.org/rfc/rfc5426 It states:

[3.1]One Message Per Datagram

   Each syslog UDP datagram MUST contain only one syslog message, which
   MAY be complete or truncated.  The message MUST be formatted and
   truncated according to [RFC 5424](https://www.rfc-editor.org/rfc/rfc5424) [[2](https://www.rfc-editor.org/rfc/rfc5426#ref-2)].  Additional data MUST NOT be
   present in the datagram payload.

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[elasticroentgen]

Any updates or workarounds for this? i have the same issue.

[rarrr]

I had to put syslog-ng in front of it and use that to convert the data into alloys version of RFC5424

[elasticroentgen]

switched to vector.dev now.