Q/BUG: cannot disable Ubuntu kernel's audit system via auditctl

[sfbahr]

I'm running a fleet of Linux hosts with Red Canary Linux EDR (Endpoint Detection and Response) which uses eBPF for gathering telemetry in service cfsvcd.service. In an older configuration, it gathered data from the kernel's audit system and everything was fine. However, when we switched cfsvcd to gathering data from eBPF instead, we noticed that the kernel ring buffer was flooded with audit messages. This is because cfsvcd.service now stops auditd.service, but leaves the kernel audit system enabled.

I've mitigated this issue by manually running # auditctl -e 0 on our hosts (via Puppet). However, I'm running into a strange issue where some hosts (~0.5%) are still logging all audit events to the kernel ring buffer even after I have disabled the audit system via # auditctl -e 0. A # auditctl -s run shows enabled 0, yet audit logs continue to flood the kernel ring buffer.

I'm running Linux kernel 5.4.0-1063-aws on Ubuntu 18.04 with auditctl v2.8.2. systemd-journald-audit.socket is masked & inactive, auditd.service is disabled & inactive.

I cannot entirely disable the audit system via a kernel parameter because Red Canary Linux EDR fails to start cfsvcd.service as it fails to read the audit netlink due to no audit support in the kernel (it shouldn't fail here, but that's a separate issue):

Jun 28 20:41:04 systemd[1]: Started Canary Forwarder Service. Jun 28 20:41:04 cfsvcd[105781]: Found config file at path '"/opt/redcanary/config.json"', continuing execution... Jun 28 20:41:12 cfsvcd[105781]: component: Sentry Jun 28 20:41:12 cfsvcd[105781]: Jun 28 20:41:12.055 INFO Initializing Sentry Config { version: "1.4.17.release.[...]", https_proxy: None, extra_tags: [("telemetry.source", "eBPF")], logging_enabled: true, metrics_enabled: true, [...]} Jun 28 20:41:12 cfsvcd[105781]: Failed to setup or configure host system: Linux Audit watcher failure: Netlink Error: IO Error: Protocol not supported (os error 93) Jun 28 20:41:12 cfsvcd[105781]: Received stop. Exiting unix::wait_for_signal. Signal: 0, done.is_signaled(): true Jun 28 20:41:12 cfsvcd[105781]: Joining async_agent_thread Jun 28 20:41:13 cfsvcd[105781]: Attempting to stop subscriber (Ebpf). Jun 28 20:41:13 cfsvcd[105781]: Shutting down due to termination signal [...] Jun 28 20:41:15 cfsvcd[105781]: Subscriber stopped. Jun 28 20:41:15 auditctl[106733]: Error - audit support not in kernel Jun 28 20:41:15 auditctl[106733]: Cannot open netlink audit socket Jun 28 20:41:15 systemd[1]: cfsvcd.service: Service hold-off time over, scheduling restart. Jun 28 20:41:15 systemd[1]: cfsvcd.service: Scheduled restart job, restart counter is at 301. Jun 28 20:41:15 systemd[1]: Stopped Canary Forwarder Service.

Here's the unit file for cfsvcd.service:

[Unit]
Description=Canary Forwarder Service

DefaultDependencies=no
After=local-fs.target systemd-tmpfiles-setup.service auditd.service
Before=sysinit.target shutdown.target
# Replace the auditd service if it is running
Conflicts=auditd.service shutdown.target

[Service]
WorkingDirectory=/opt/redcanary
ExecStart=/opt/redcanary/cfsvcd
ExecStopPost=-/sbin/auditctl -D
Restart=always
TimeoutSec=15

[Install]
WantedBy=multi-user.target

Is this a known issue? Is there a workaround to stop the logging to the kernel ring buffer? Is there any more information I can provide to help debug?

I originally emailed this to linux-audit@redhat.com ~a week ago but looks it's still waiting on moderator approval. I can follow up here or there, whichever is preferred.

[pcmoore]

I originally emailed this to linux-audit@redhat.com ~a week ago but looks it's still waiting on moderator approval. I can follow up here or there, whichever is preferred.

A quick note on that, due to moderation problems, development of the upstream Linux Kernel has moved to a new list:

• http://vger.kernel.org/vger-lists.html#audit
• https://github.com/linux-audit/audit-kernel/blob/main/MAINTAINERS#L3373

[pcmoore]

Any chance you can reproduce the problem on an unpatched, upstream Linux Kernel without any eBPF programs loaded? I ask simply because it can be difficult for us to diagnose problems when we don't know what is going on in the kernel; distro patches and eBPF programs can often have unintended consequences.

Normally I would suggest adding audit=0 to the kernel command line to disable auditing at the earliest stages of boot, as you've already tried, but it looks like that would cause your eBPF program to fail. At this point I would suggest following up with Red Canary, possibly AWS as they are providing your kernel build. I'm not sure where you are disabling audit, but you could try disabling audit earlier in the boot if possible. There was also a point in time where systemd unconditionally enabled audit, regardless of the audit configuration; I'm not sure if that it still the case, but that might be another avenue to pursue.

[pcmoore]

I'm going to close this out right now as the mix of a distro kernel and a third-party eBPF addon means there isn't much we can do beyond the comments above. If you have any additional information, comments, etc. feel free to reopen the issue, but right now I think the best we can do as upstream developers is wish you luck :)