[EKS] [Audit log]: Allow to custom audit-log policy

haupv-bkt commented 4 years ago

Tell us about your request Currently, we could only enable audit log with the default policy. Is it possible to ship audit-log-policy to config map for customization?

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Currently, EKS use audit-log with default policy (log all request via kube-api), it causes a huge log to process on our system (most of these logs come from trusted local service). Could we custom audit policy to reduce unnecessary logs?

lapidus79 commented 4 years ago

This is kind of important, since the audit log costs can compound quickly in cloud watch.

vijayc08 commented 4 years ago

Do we have any plan to allow EKS logging policy customisation for EKS users ?

tmatias commented 3 years ago

Currently, some audit logs do spit JWT tokens. At work (Nubank), we use centralized logging, and generally like and see value in making them available to everyone (audit logs, for example, do find casual audiences, in the form of "oh, weird that the HPA is that way, I guess if I can find who changed it maybe I can get context as to why"). The fixed policy is blocking us from doing that (which frankly, seems like a gap for us, which used to be possible running our own distro), while a custom policy would do so (as we could omit them via the policy).

saleem-mirza commented 3 years ago

EKS team please, this request begs for your attention!

RicardsRikmanis commented 2 years ago

I'm betting a lot of people come to this Github issue after seeing their Cloudwatch costs when enabling audit logging.

I'm echoing everyone else, please let us customize audit policy.

inboxamitraj commented 2 years ago

+1 for this issue.

pwen commented 2 years ago

+1

awoimbee commented 2 years ago

I'm betting a lot of people come to this Github issue after seeing their Cloudwatch costs when enabling audit logging.

I'm one of these people. Does anyone have tips & tricks to partially solve this issue ? On a simple dev cluster with only audit logging I end up with a 40$ CloudWatch bill. And I don't even want to use CloudWatch, I have my own logging system in the cluster !

saleem-mirza commented 2 years ago

I'll suggest to write few lines about why custom audit police or even log delivery to S3 is important and if lack of this feature is impacting your adoption. It will help EKS team to prioritize the request.

pankaj405586 commented 2 years ago

By when we can expect custom audit log policy in EKS ? cloud watch billing is about to touch sky.

jtackaberry commented 2 years ago

One of the services I support recently had a misbehaving application in staging that was beating on the K8s API (list pods and nodes) and was burning $800/day on Cloudwatch alone for several days until it tripped a billing alert. Suffice it to say we immediately disabled control plane audit logging until the application was fixed. Then we had some 'splaining to do.

This should have been entirely avoidable. I don't care about auditing read-only calls, and would have filtered those out had I had the choice.

S3ky commented 2 years ago

I agree with @jtackaberry. Ability to custom audit-log policy can bring us additonal cost savings. +1

eladleev-nate commented 2 years ago

+1

Apurv11 commented 1 year ago

+1

pescador691 commented 1 year ago

I also would like to see the ability to custom the audit log policy to bring my team an additional cost savings. +1

fubarhouse commented 1 year ago

Completely agree with the above comments, but this could be potentially provided in addition to, or as an alternative to this one. It aims to solve the same problem, even if it provides less flexibility.

https://github.com/aws/containers-roadmap/issues/1141

You can pass a file with the policy to kube-apiserver using the --audit-policy-file flag. If the flag is omitted, no events are logged. Note that the rules field must be provided in the audit policy file. A policy with no (0) rules is treated as illegal.

https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/

andrewnicolalde commented 1 year ago

It's extremely disappointing that this isn't yet supported. I'm beginning to suspect the issue on AWS's end is to do with GuardDuty requiring (specific?) audit logs from the control plane to perform EKS detections..?

mikestef9 commented 1 year ago

How important is fully customizable audit log here, vs if EKS introduced a new minimal audit log type, for example that excluded get and list requests from the logs?

saleem-mirza commented 1 year ago

@mikestef9 in my humble opinion, let the users have option and make an informed decision. Get and List requests do have significant role to play especially in security arena. Although, I know the implications of a fully bloated logs, but again, let use make an informed decision.

sercasti commented 1 year ago

How important is fully customizable audit log here, vs if EKS introduced a new minimal audit log type, for example that excluded get and list requests from the logs?

Sorry to be blunt, but after 4 years, anything will do, the sooner the better, minimal audit type will save thousands of $ in cloudwatch logging, which is the main driver for this ask

NathanFRuiz commented 11 months ago

It's concerning how long has it been with no answer, audit logs are mandatory to be compliant with some standards (such as PCI), so some companies can't simply turn it off, other than that since my company uses a external SIEM tool we'll have to work around filtering the logs from cloudwatch before shipping them, otherwise those logs by themselves will burn all our daily quota in just a few hours

ClaytonOlleyNutrien commented 11 months ago

I believe AWS recently announced that these EKS logs are at least "vended" in Cloudwatch now so they should be considerably less expensive.

saleem-mirza commented 11 months ago

Well, less expensive is not a solution. Either AWS make it completely free or empower customers to export logs wherever they want.

chlunde commented 10 months ago

@mikestef9 That would help, but I would like Secret read/list audited too, and as others suggest, they have their own opinions :)

yardenw-terasky commented 8 months ago

It could be beneficial to get that feature, Costs are extraordinary when every log is collected. A customization that allows you to choose which logs specifically to collect would solve that.

From Kubernetes docs, it seems like it's customizable. https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/

EmrhT commented 8 months ago

It's not reasonable for this issue to last for 4 years. Audit logs, which is a native feature of K8S control plane, should be fully customizable according to customers' needs. Hundreds of upvotes above shouldn't be meaning much for AWS. In the meantime, making EKS cloudwatch logs (only audit ones) free might cause customer satisfaction at least.