Audit to splunk

[mreiger]

Pass a config file to the auditforwarder fluent-bit that forwards copies of all audit events to splunk. (Also set kube-apiserver service externalTrafficPolicy to local so that the clients' real ip addresses appear in the audit events.)

[majst01]

Why do we need another audit forwarding to a very specific implementation of log backend. Does the actual log forwarding not work, and if not why not fixing this

[mreiger]

The mechanism for forwarding the audit data to splunk was changed: Now we pass a config file to the fluent-bit in the auditforwarder that sends copies of the audit events directly from the seed to splunk. We do need the splunk connection because of compliance requirements for our gardener-operated clusters.

[majst01]

After some internal discussion i would like to clarify the following aspects:

• logging to splunk should be configurable per shoot in terms of:
  • enabled/disabled
  • target host
  • credentials
• this must be possible also for already created shoots
• by default we should only consider prod clusters (aka purpose)
• what happens with the logs if the splunk endpoint is not reachable? Spool to local PV or drop these events. If spooling how much space should be held for spooling ? What happens to audit logs if spooling area is full?

[mreiger]

After some internal discussion i would like to clarify the following aspects:

* logging to splunk should be configurable per shoot in terms of:

  * enabled/disabled
  * target host
  * credentials

* this must be possible also for already created shoots

* by default we should only consider prod clusters (aka purpose)

Current state of discussion:

• The clusterAudit featureGate defaults to true, for all clusters.
• The auditToSplunk featureGate defaults to false, and can be enabled manually for the clusters where it is required
• There is only one splunk endpoint and index, defined in the controller registration, for all clusters where logging to splunk via featureGate is enabled
• Logging to different endpoints / indices can be achieved by:
  • Not enabling the auditToSplunk featureGate
  • Deploying either the splunk connect for kubernetes helm chart or a custom daemonset into a cluster that will pick up the audit data from the audittailer pod

* what happens with the logs if the splunk endpoint is not reachable? Spool to local PV or drop these events. If spooling how much space should be held for spooling ? What happens to audit logs if spooling area is full?

Auditforwarder buffers into memory; the current version has implemented a memory buffer limit; once this is full, further audit events will be dropped.

The default memory buffer limit is 200 Mbyte which is enough for some hours of log files.

Also there auditforwarder container now gets deployed with limits so that it can not grow indefinitely anyway.

I hope this addresses most concerns.

[Gerrit91]

Today we had a discussion on how we could achieve logging on a per-shoot configuration basis + falling back to a default audit logging. I want to keep these ideas for future reference.

Technically possible:

• Multiple splunk configurations in GEPM for tenants, with fallback default to FITS splunk
  • (-) not per shoot, only per tenant, requires deployment in change window (+) easy to implement, not hacky
• Create k8s client from gardenlet kubeconfig secret and read custom config from gardener-apiserver, deploy audit configuration secret through cloud-api
  • (-) hacky / kind of dangerous, (+) self-service, custom per shoot, not exposing any information in shoot spec
• Add audit configuration to shoot spec (control plane provider config), configure through cloud-api
  • (-) Exposes secrets in the shoot spec, which is against Gardener objectives, (+) self-service, custom per shoot, relatively ease to implement
• User deploys splunk configuration to certain namespace (audit) in his shoot cluster, GEPM picks up the secret and starts logging, default config provided through GEPM controller registration
  • (-) Needs special documentation how to use, does not work on first reconcile (+) custom per shoot, self-service

Not possible:

• Gardener audit policy mechanism: Only policy file, webhook configuration cannot be provided through a user interface

[mreiger]

Record of the decisions made today:

• clusterAudit (turning on auditing and forwarding the log into the cluster) will get turned on by default for new clusters in cloud-api/cloudctl. For existing clusters it will have to be explicitly turned on, and default to false otherwise.
• auditToSplunk is turned off by default
  • If turned on, by default it will log into a default splunk endpoint provided through the controller-registration.
  • If the cluster user provides a different splunk endpoint, through a secret in the audit namespace, it will be used instead of the default one. This is variant 4 in the comment above. (But the default splunk endpoint is not written to the shoot, to prevent leaking of credentials.)