Auditbeat 'operation not permitted', even with highly elevated container

[jarpy]

The Auditbeat image currently fails with 'operation not permitted' even when:

• The container process runs as root
• The container is started with --privileged
• The container is granted all capabilities (--cap-add=ALL)

# docker run --privileged -u root --cap-add=ALL --rm -it docker.elastic.co/beats/auditbeat:6.0.0-beta2
2017/09/06 04:16:05.881553 beat.go:430: INFO Home path: [/usr/share/auditbeat] Config path: [/usr/share/auditbeat] Data path: [/usr/share/auditbeat/data] Logs path: [/usr/share/auditbeat/logs]
2017/09/06 04:16:05.881593 beat.go:457: INFO Beat metadata path: /usr/share/auditbeat/data/meta.json
2017/09/06 04:16:05.881623 metrics.go:23: INFO Metrics logging every 30s
2017/09/06 04:16:05.881915 beat.go:437: INFO Beat UUID: a1c35d76-0446-4282-93b8-4c9f8d17750f
2017/09/06 04:16:05.881941 beat.go:192: INFO Setup Beat: auditbeat; Version: 6.0.0-beta2
2017/09/06 04:16:05.882124 client.go:123: INFO Elasticsearch url: http://elasticsearch:9200
2017/09/06 04:16:05.882479 module.go:80: INFO Publisher name: 9faf05cdf86d
2017/09/06 04:16:05.882515 metricbeat.go:37: INFO Register [ModuleFactory:[], MetricSetFactory:[audit/file, audit/kernel]]
2017/09/06 04:16:05.882670 cfgwarn.go:23: WARN EXPERIMENTAL: The audit.kernel metricset is a beta feature
2017/09/06 04:16:05.883142 cfgwarn.go:23: WARN EXPERIMENTAL: The audit.file metricset is an experimental feature
2017/09/06 04:16:05.883347 beat.go:264: INFO auditbeat start running.
2017/09/06 04:16:13.740069 audit_linux.go:81: ERR [audit.kernel] failed to delete existing rules: operation not permitted

Very interesting. I feel like there's something I don't know about audit permissions. Any thoughts @andrewkroh?

[jarpy]

Perhaps I just need to disable auditd.

EDIT: Nope, it's not even installed.

[andrewkroh]

The kernel checks that the client

• is in the initial user and pid namespaces
• has the CAP_AUDIT_CONTROL capability (when Auditbeat is operating as the primary userspace audit daemon)
• has the CAP_AUDIT_READ capability (when Auditbeat is only listening to events on a multicast socket) [this is a new feature that hasn't merged yet and requires kernel 3.16+].

So adding --pid=host --cap-add=AUDIT_CONTROL --cap-add=AUDIT_READ should be sufficient to make Auditbeat work in a container.

[jarpy]

Thanks! --pid=host was, indeed, the required magic.

I started out with the capabilities you mentioned since I assumed they were the needed ones. After that, I added progressively more desperate (and insecure) options in a vain attempt to resolve the issue.

Confirming the required capabilities raises another interesting challenge, that you will "appreciate", @dliappis. We can't easily set CAP_AUDIT_READ on the MetricbeatAuditbeat binary while buiding the image. Why? Because CAP_AUDIT_READ was introduced in Linux 3.16, but the setcap we have in Centos 7 was compiled with capability.h from Linux 3.10. Nasty.