Syslog Input: expecting a MSGLEN

[mthota15]

is this issue #4335 fixed?

I am seeing same errors in latest version of telegraf(1.7.2). I am trying to send my docker daemon logs via syslog drivers.

018/07/27 16:09:10 I! Using config file: /etc/telegraf/telegraf.conf 2018-07-27T16:09:10Z I! Starting Telegraf v1.7.2 2018-07-27T16:09:10Z I! Loaded inputs: inputs.syslog 2018-07-27T16:09:10Z I! Loaded aggregators: 2018-07-27T16:09:10Z I! Loaded processors: 2018-07-27T16:09:10Z I! Loaded outputs: file 2018-07-27T16:09:10Z I! Tags enabled: 2018-07-27T16:09:10Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"", Flush Interval:10s 2018-07-27T16:09:11Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN 2018-07-27T16:09:16Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN 2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN 2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN 2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN 2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN

Here is my docker daemon json config:

{
  "log-opts": {
    "syslog-address": "tcp://localhost:6514"
  },
  "debug": true,
  "experimental": false,
  "log-driver": "syslog"
}

[glinton]

I'll double check the way docker syslog works, but I imagine we'll just need a new format to parse logs as

[glinton]

Thanks for bringing this up. It appears we are always parsing syslogs as if they were sent TLS style (RFC5425) with a MSG-LEN. You can configure telegraf with a tls_[key|cert] pair in the meantime and configure docker to send tcp+tls. Also note you'll have to specify the syslog-format to be rfc5424micro

telegraf.conf

[[inputs.syslog]]
  server = "tcp://localhost:6514"
  tls_cert = "/tmp/thing.crt"
  tls_key = "/tmp/thing.key"

daemon.json

{
  "log-opts": {
    "syslog-tls-skip-verify": "true",
    "syslog-address": "tcp+tls://localhost:6514",
    "syslog-format": "rfc5424micro"
  },
  "log-driver": "syslog"
}

Last note: you won't have to include syslog-tls-skip-verify if telegraf syslog plugin is using valid certs.

[danielnelson]

The UDP version works without requiring any certificates:

telegraf.conf

[[inputs.syslog]]
  server = "udp://localhost:6514"

daemon.json

{
    "log-driver": "syslog",
    "log-opts": {
        "syslog-address": "udp://localhost:6514",
        "syslog-format": "rfc5424micro"
    }
}

@leodido Should we select between the 5424 and 5425 parsers depending on if the TCP socket is configured for TLS?

[leodido]

RFC5424 does not explicitly address the type of framing.

Unlike RFC5425, which explicitly provides octet counting - ie., MSGLEN. It is not a matter of protocol (TLS, TCP, UDP) but of RFC5425 specification.

So my answer is no, we should not select the parsers depending on the protocol.

RFC5425 does not contemplate another framing technique.

It has been done exactly for this reason, since RFC5424 was missing this part and various client started, in the past, to make custom framing techniques.

Please notice that octet counting allows syslog message to be split across different packets, among other things.

Therefore, since we are providing RFC5425 parsing respecting this RFC, ideally, we should not modify this behavior.

Solution

What could be done, imho, is to provide another parser respecting the RFC5424 (we already have it and we use it internally in RFC5425) leaving the choice of the framing technique (eg., old one like new lines and so on) to the user.

But imho it should be enabled via options by the user, not switched depending on the protocol in use.

This way users can also use clients not respecting RFC5425 and implementing custom framing techniques.

[glinton]

I agree that we could expose some options to configure the spec followed as I think we would run into issues by simply checking if the plugin is configured with tls prior to parsing. For example rsyslog can prefix the message length to the log by adding (o) to the entry in rsyslog.conf so even though RSYSLOG_SyslogProtocol23Format (RFC5424) is specified, the message can still be transmitted to match RFC5425.

---

Examples:

Config with no (o):

*.* @@127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Produces (RFC5424 output):

<30>1 2018-08-01T11:16:08.529324-06:00 hilldale systemd 1 - -  Started System Logging Service.

Config with (o):

*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Produces (RFC5425 output):

95 <30>1 2018-08-01T11:12:29.276656-06:00 hilldale systemd 1 - -  Started System Logging Service.

[leodido]

That's exactly what I am saying. It is not protocol dependant.

RFC5425 basically wraps RFC5424 with octet counting framing technique.

An user can utilise a syslog relay which emits RFC5424 (ef., rsyslog default behavior).

On Wed, Aug 1, 2018, 7:28 PM Greg notifications@github.com wrote:

I agree that we could expose some options to configure the spec followed as I think we would run into issues by simply checking if the plugin is configured with tls prior to parsing. For example rsyslog can prefix the message length to the log by adding (o) to the entry in rsyslog.conf so even though RSYSLOG_SyslogProtocol23Format (RFC5424) is specified, the message can still be transmitted to match RFC5425.

Examples:

Config with no (o):

. @@127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Produces (RFC5424 output):

<30>1 2018-08-01T11:16:08.529324-06:00 hilldale systemd 1 - - Started System Logging Service. Config with (o): *.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format Produces (RFC5425 output): 95 <30>1 2018-08-01T11:12:29.276656-06:00 hilldale systemd 1 - - Started System Logging Service. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

[glinton]

I mean the RFC5425 spec title and such could be more clear though; "Transport Layer Security (TLS) Transport Mapping for Syslog." Plus, it does state:

All syslog messages MUST be sent as TLS "application data"

Took some digging into rsyslog configuration to realize why it was sending the MSG-LEN

[danielnelson]

If we add a boolean option: octet_framing = true, I wonder if it should be valid in all protocols. Do syslog writers support non-octet framing in TLS? Would anyone combine octet-framing with UDP?

This would also mean one additional option that must be set before using the plugin, perhaps we should it as a multi value option instead: octet_framing = "enabled|disable|auto"

[JulienChampseix]

Hi @danielnelson @glinton, We are getting same issue @mthota15 Is there any progress on it cuz it can block our POC using influxdb ? (versus ES + Kibana)

[danielnelson]

@JulienChampseix We haven't had time to work on this yet, it is probably something for 1.9.0. I think we should do the 3-state option:

octet_framing = "enable|disable|auto"

Default would be auto, and enable/disable would just force using either the rfc5424(disable) or the rfc5425(enable) parser into use.

[russorat]

blocked by https://github.com/influxdata/go-syslog/issues/17

[FrankyBoy]

So can someone actually concisely TL;DR this whole mess in how to configure rsyslog and telegraf to play nice with each other?

[russorat]

@FrankyBoy were you able to follow the steps in the README? https://github.com/influxdata/telegraf/tree/master/plugins/inputs/syslog#rsyslog-integration

If something is missing there, let's open an issue to update the docs.

[FrankyBoy]

Yes, and the udp based path definitely doesn't work right now for me. TCP is not an option for various reasons.

[Sieboldianus]

I have the same problem with the Keycloak Syslog plugin, which follows RFC 5424 but also leads to MSGLEN error.

2024-08-19T10:45:36Z E! [inputs.syslog] Error in plugin: found ILLEGAL(<), expecting a MSGLEN

Telegraf/influxdb work if I manually compose a test log entry with MSGLEN:

echo "57 <13>1 2018-10-01T12:00:00.0Z example.org root - - - test" | nc 127.0.0.1 6514

I cannot change the way the Keycloak syslog sends these entries.

[edit] Doh!

It is writting in the docs:

The syslog plugin listens for syslog messages transmitted over a Unix Domain socket, UDP, TCP, or TLS; with or without the octet counting framing.

Keycloak syslog requires without the octet counting framing. And in the telegraf plugin it says:

>  ## Framing technique used for messages transport
>  ## Available settings are:
>  ##   octet-counting  -- see RFC5425#section-4.3.1 and RFC6587#section-3.4.1
>  ##   non-transparent -- see RFC6587#section-3.4.2
>  # framing = "octet-counting"

I only had to enable non-transparent framing in the Telegraf syslog plugin:

framing = "non-transparent"