Auditing of Root CA Key access

[gawertm]

As a short term to mitigate the risk of the root CA private key being stored on control plane nodes of the workload clusters, we need to look into auditing:

• Define resources that need to be audited
• Set up alerting based on the audit logs
• Investigate the use of Falco for auditing
• Explore auditing capabilities introduced by Atlas

[vvondruska]

Tested auditing of the content of /etc/kubernetes/pki with Falco 0.36. It work, although the logs are only printed in the standard output of the Falco pods. Giantswarm offers Falco as an installable app, but the version of Falco in this app is 0.35.x, which is missing convenience selectors to define rules that deal with file system. Defining rules to audit access to files before v0.36 is much more complicated according to the documentation.

Next steps:

• talk to team Shield and ask if it's possible to release a new version of the Falco app with v0.36
• decide whether we want to make Falco a default app if we want to use it for auditing of the root CA key file
• figure out a way to export the audit logs and create alerts based on them
• fine-tune the rules for auditing

[vvondruska]

Falco app v0.7.0 can be used to audit accesses to the root CA private key in control places. Falco Exporter can expose the audit logs as metrics to Prometheus.

However, in order to achieve this both Falco and Falco Exporter would have to be installed on all clusters and they would have to be configured to run on control planes.

Falco Exporter is not able to run due to the current security constraints defined in the Falco app.

[vvondruska]

Reached out to Team Shield with some questions related to the above: https://gigantic.slack.com/archives/C02FL4EAADD/p1701337202749149

[vvondruska]

Feedback from Team Shield: In general it is OK to run Falco and Falco Exporter on control plane nodes and use them to audit access to the root CA key. However, a few concerns were raised:

• Falco currently runs on MCs, its rules are not sanitized for WCs - it may generate too much noise and have an impact on performance
• Running Falco in general may have an impact on performance
• Security constraints need to be adjusted to allow Falco Exporter to run

[vvondruska]

Next steps:

• Test performance of Falco with sanitized rules for auditing access to root CA private key on a testing WC
• Once completed, run the same test on a production WC in Gazelle

Use a fresh installation of Falco without the unsanitized default rules defined in the Falco GS app

[vvondruska]

Performance:

Falco has specific resource requests and limits defined. It requests 0.1 CPU and 512 Mi of memory, and the limit is set to 1 CPU and 1 Gi of memory.

Falco runs a pod on each node in the cluster, so the resource requests and limits also apply to each individual node.

Falco Exporter does not define any resource requests or limits by default. However, in the default values it provides a suggested default resource requests of 0.1 CPU and 120 Mi of memory.

During tests in testing WCs the observed resource consumption was lower than the limit or the request.

[vvondruska]

Successfully tested the use of Falco Exporter to expose logs to Prometheus.

Log messages from Falco appeared in Prometheus running on the MC and can be used to fire an alert.

However, many details from the log messages were omitted in the exposed Prometheus metrics. It would be nice to have the additional details because they can be used to filter out legitimate uses of the root CA key and prevent the alert from firing in such cases.

There may be other ways to expose the logs from Falco with full details and configure an alert based on them. One of these ways is the observability bundle and Loki. Team Atlas currently works on a solution, which will allow any app installed in the WCs to send its logs to Loki and make them available in our Grafana. Once this solution is ready, we can check it out and see if it is suitable for our use case.

[vvondruska]

Finished testing the initial rule definition. Reached out to Team Shield to determine what is the best way to add the rule to the Falco app, and concluded that it should be defined in the default values of the app. Opened a PR in falco-app, which adds the rule to the default values: https://github.com/giantswarm/falco-app/pull/261

[vvondruska]

A new rule, which enables Falco to log access to the root CA key file has been merged to the Falco App. Next steps:

• release Falco
• make Falco enabled by default in the Security bundle

[tuladhar]

Follow-up

• 👍 Falco v0.9.0 released with rule to detect access to root CA key.
• 👎 As of security-bundle v1.8.1, falco is not yet enabled by default.