[FR] Debian 12+ rsyslog parsing Implementation

[Aegrah]

Issue

From Debian 12 and up, Debian switched to journalctl and no longer ships with default syslog. The go-to syslog implementation for Debian is the rsyslog package, downloadable from the APT repositories. This implementation of syslog has different mappings than the "default" syslog on earlier versions of Debian. Thus, it doesn't have several essential ECS fields, making several Linux detection rules no longer function.

rsyslog: https://www.rsyslog.com/

Examples

2024-07-11T11:30:06.943880+02:00 hostname useradd[14155]: new group: name=user, GID=1001
2024-07-11T11:30:06.958217+02:00 hostname useradd[14155]: new user: name=user, UID=1001, GID=1001, home=/home/user, shell=/bin/sh, from=/dev/pts/3

Several unusual fields include: system.auth.useradd.home system.auth.useradd.shell

Additionally, the parsed logs no longer contain an event.category.

Request

This issue will likely exist for Debian 13 when it gets released as well. We might want to consider parsing these in a way that allows us to stay consistent with ECS and other syslog versions.

Example full log:

{
  "_index": ".ds-logs-system.auth-default-2024.07.07-000036",
  "_id": "V1CDoZABZpyWyvSVExVX",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "",
      "id": "",
      "ephemeral_id": "",
      "type": "filebeat",
      "version": "8.13.4"
    },
    "log": {
      "file": {
        "path": "/var/log/auth.log"
      },
      "offset": 5721,
      "syslog": {
        "hostname": "debian-12",
        "appname": "useradd",
        "procid": "4235"
      }
    },
    "elastic_agent": {
      "id": "",
      "version": "8.13.4",
      "snapshot": false
    },
    "tags": [
      "system-auth"
    ],
    "input": {
      "type": "log"
    },
    "@timestamp": "2024-07-11T10:57:12.000Z",
    "system": {
      "auth": {
        "useradd": {
          "shell": "/bin/sh, from=/dev/pts/1",
          "home": "/home/testuser"
        }
      }
    },
    "ecs": {
      "version": "8.0.0"
    },
    "related": {
      "hosts": [
        "debian-12"
      ],
      "user": [
        "testuser"
      ]
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "system.auth"
    },
    "host": {
      "hostname": "debian-12",
      "os": {
        "kernel": "6.1.0-9-amd64",
        "codename": "bookworm",
        "name": "Debian GNU/Linux",
        "family": "debian",
        "type": "linux",
        "version": "12 (bookworm)",
        "platform": "debian"
      },
      "containerized": false,
      "ip": [
        "",
        ""
      ],
      "name": "debian-12",
      "id": "",
      "mac": [
        ""
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2024-07-11T11:17:20Z",
      "timezone": "+02:00",
      "kind": "event",
      "dataset": "system.auth"
    },
    "user": {
      "name": "testuser",
      "id": "1001"
    },
    "group": {
      "id": "1001"
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.13.4"
    ],
    "host.os.name.text": [
      "Debian GNU/Linux"
    ],
    "host.hostname": [
      ""
    ],
    "host.mac": [
      ""
    ],
    "agent.name.text": [
      "debian-12"
    ],
    "host.os.version": [
      "12 (bookworm)"
    ],
    "host.os.name": [
      "Debian GNU/Linux"
    ],
    "agent.name": [
      "debian-12"
    ],
    "host.name": [
      "debian-12"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "event.kind": [
      "event"
    ],
    "host.os.type": [
      "linux"
    ],
    "user.id": [
      "1001"
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      5721
    ],
    "data_stream.type": [
      "logs"
    ],
    "related.user": [
      "testuser"
    ],
    "tags": [
      "system-auth"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "system.auth.useradd.shell": [
      "/bin/sh, from=/dev/pts/1"
    ],
    "agent.id": [
      ""
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "host.containerized": [
      false
    ],
    "agent.version": [
      "8.13.4"
    ],
    "related.hosts": [
      "debian-12"
    ],
    "host.os.family": [
      "debian"
    ],
    "log.syslog.hostname": [
      "debian-12"
    ],
    "group.id": [
      "1001"
    ],
    "log.syslog.appname": [
      "useradd"
    ],
    "user.name": [
      "testuser"
    ],
    "host.ip": [
      "",
      ""
    ],
    "agent.type": [
      "filebeat"
    ],
    "event.module": [
      "system"
    ],
    "host.os.kernel": [
      "6.1.0-9-amd64"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "host.id": [
      ""
    ],
    "event.timezone": [
      "+02:00"
    ],
    "system.auth.useradd.home": [
      "/home/testuser"
    ],
    "elastic_agent.id": [
      ""
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "bookworm"
    ],
    "log.syslog.procid": [
      "4235"
    ],
    "event.ingested": [
      "2024-07-11T11:17:20.000Z"
    ],
    "@timestamp": [
      "2024-07-11T10:57:12.000Z"
    ],
    "host.os.platform": [
      "debian"
    ],
    "data_stream.dataset": [
      "system.auth"
    ],
    "log.file.path": [
      "/var/log/auth.log"
    ],
    "agent.ephemeral_id": [
      "17257fa2-37e3-4dea-b714-240ec1399341"
    ],
    "event.dataset": [
      "system.auth"
    ],
    "user.name.text": [
      "testuser"
    ]
  }
}

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[nick-alayil]

@Aegrah, which modules and processors are you expecting for this enhancement? I noticed there's a Filebeat syslog module, syslog processor and a logstash syslog plugin. I'm ruling out the latter. Any thoughts or priorities here?

Btw, I don't see Deb12 listed as a supported OS for agent and filebeat in the support matrix.

Updating the project and team tags based on the ownership details provided below:

https://github.com/elastic/beats/blob/5d44853136c47038c2a75ffd11f7c57982b188fa/.github/CODEOWNERS#L31 https://github.com/elastic/beats/blob/5d44853136c47038c2a75ffd11f7c57982b188fa/.github/CODEOWNERS#L64

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[Danouchka]

Hi @nick-alayil , I am not at that low level. All started with an Elastic Agent trying to bring audit logs , system logs from Debian 12 and enable elastic security rules

[Aegrah]

@nick-alayil, Seeing that @Danouchka mentioned the Elastic Agent, that would mean that the customer experienced this issue when using the Agent's syslog processor. But you are correct to state that we indeed do not support Debian 12, I did not know that.

If we do not support Debian 12 in general, and have no plans in supporting it in the future, adding support for rsyslog does not make much sense, as we are currently only experiencing this issue with the Debian 12 rsyslog implementation. I will let the next assignee from Security-Deployment and Devices decide on whether we close this out, or go for an implementation.

Thank you!

[nick-alayil]

My recommendation is to close this ticket and raise an enhancement request to add Debian 12 support for the Elastic Agent. Once this support is added, the issue should ideally be resolved. Additionally, I am unsure if this use case is part of the test case that qualifies agent support on an OS. Therefore, I recommend mentioning this ticket in the ER.

[Aegrah]

I will close out this issue, as it can now be tracked from https://github.com/elastic/enhancements/issues/22255.