Audit checks RFC.

[jvanz]

Adds the RFC to discuss how to implement the audit checks.

[jvanz]

After creating this draft PR I've realized that I missed to add a text about how the audit checks will work with multiple policy servers. I'll add that before marking it as ready for review.

[raulcabello]

This is looking great @jvanz, thanks! I left some ideas

[viccuad]

As talked in planning, moving to blocked to get feedback from Flavio.

[flavio]

I've read through the comments, this is an overview of how I feel about this topic.

Leveraging Kubernetes' dry-run feature

Using the dry-run feature of the API server was a clever idea, but I don't think this is going to work in our scenario.

Let's say we have two policies that are evaluating pods: polA and polB. Submitting a dry-run of an existing pod results in a failure. However, it would be impossible for us to know which policy actually rejected the pod. Maybe it was polA, maybe it was polB, maybe both of them are not fine with it.

Because of that, I think the only way is to operate in a different way.

The guiding principle

What's important for us is to come up with a solution that has a rock solid algorithm. We have to come up with a series of steps that receives some input data and produces output based on that. The last step of the workflow will lead to the creation of the PolicyReport.

How the input data is obtained can be changed over the time. The goal is to have something we can ship and analyze against real world data.

Maybe in the beginning there are going to be limitations about what kind of scenarios we are going to support, but these should be overcome in the future without having to rewrite big parts of the code.

Let's focus on Namespaces

I think Namespace are the minimal computational unit of the audit reports. We should define the steps needed to create the PolicyReport for one specific Namespace. Once this is done, the actual code can be run in parallel or in sequential mode against all the Namespaces, this doesn't matter too much.

Finally, the process to create the PolicyReport for a single Namespace, can be applied also to generate the ClusterPolicyReport, which is the one targeting only cluster-wide resources.

The following sections focus on each step of the process.

Step 1: find relevant policies

Given a Namespace to be audited, create a list of all the policies that are interested about the Namespace.

All the AdmissionPolicy resources that are defined inside of the namespace are part of the list.

For ClusterAdmissionPolicy resources, the Namespace selector has to be used to figure out if the audited Namespace is relevant.

Note: we will consider only policies that inspect CREATE events

Step 2: find the relevant Kubernetes resources

Now we have a list of all the policies that are interested about the Namespace being audited. We iterate over each one of them and create a dictionary that has:

• Key: the Kubernetes resource type, for example Pod
• Value: a list of policies that evaluate the given Kubernetes resource

The idea is to have a map of policies that tell us these information:

• Pod resources are relevant to polA, polB
• Deployment resources are relevant to polA
• Ingress resources are relevant to polC

Step 3: fetch Kubernetes resources to be audited

Next we iterate over the keys of this dictionary and, for each one of them we query the API server to get all the resources of type X defined inside of our Namespace.

For example, given the previous example, we would end up with:

• List of all the Pod objects defined inside of the audited Namespace
• List of all the Deployment objects defined inside of the audited Namespace
• List of all the Ingress objects defined inside of the audited Namespace

Note: there's no need to do any caching of the response we get from the Kubernetes API server because of the following reasons:

• We work on a per Namespace basis, these query results are useful only in the context of the Namespace being audited
• The audit reports is generated every X minutes (maybe 10/30/60 minutes), cached data would be useless after this time

Step 4: attempt to reuse previous evaluation results

Now we have a these information:

• List of policies interested by a certain type of Kubernetes resource, like Pod
• List of Kubernetes resources, Pod in this case, defined inside of the Namespace we are auditing

The code will then retrieve the PolicyReport for the namespace being audited.

We now iterate over each resource, for example over each Pod resource defined inside of the Namespace, and perform the following operations:

• Iterate over all the policies that are interested in Pod resources, for each tuple of (policy X, pod instance Y):
  • Look into the PolicyReport for an entry that features policy X evaluating pod instance Y:
  • If there's an entry, look at the evaluation timestamp of this entry. If policy update timestamp and pod instance update timestamp are older than the evaluation timestamp it means that nothing changed. Hence we are going to reuse the old evaluation result. Otherwise something changed, hence we need to perform an evaluation.
  • If there's no entry we will perform an evaluation

Once the outer loop is done (the one against all the Kubernetes resources defined inside of the audited Namespace that are affected by policies), our code will have all the data needed to create the PolicyReport for the namespace being audited.

Step 4.5: perform evaluations

In the previous section, we saw how sometimes we need to perform a policy evaluation.

The evaluation happens by creating a fake CREATE event that has the Kubernetes object being audited as request.object.

As a first approach, I propose to perform this evaluation by making an HTTP request against the actual PolicyServer that is running the policy.

We can extend Policy Server to have a new endpoint for the audit checks. While the Kubernetes API server uses /validate/POL_ID, the audit scanner can file its requests against /audit/POL_ID.

The /audit endpoint will be different from the /validate one in these regards:

• Generate different prometheus metrics: we don't want the overall /validate metrics to be polluted by the audit checks
• Generate different Open Telemetry traces: again, these should not be mixed with the /validate ones. We might even want to turn them off to make the audit execution faster
• Policies in monitor mode should still behave like the ones in protect mode. They have to produce rejections if needed

By issuing the audit request against a real PolicyServer, we solve these problems:

• Policies can be hosted on registries secured with credentials. We don't have to worry about being able to fetch the WebAssembly modules from these authenticated registries
• Policies might be hosted on insecure registires/registires with custom TLS certificates. Again, we won't have to bother about that
• Some data used by the policy (like external OCI manifests, sigstore verification results) can be cached by the PolicyServer and reused between the audit evaluations and the regular validations requests

Of course, sending all our auditing requests to the actual PolicyServer could lead to poor performance of the regular admission requests. Currently, based on our load testing data, one single PolicyServer running with 2 workers can evaluate about 1600 requests per second. Based on this data, we think the requests generated by the audit checks should not clog the policy server. If that turns out to be true, we can embrace different strategies to mitigate that.

Until we have real world data, it's not useful to eagerly optimize this aspect.

Step 5: write PolicyReport

The code will overwrite the old PolicyReport instance with a new generated from the data kept in memory. By doing that, we don't have to worry about synchronizing the old contents with the new ones.

Reducing the scope to keep things simple

In the beginning we should reduce the scope of the audit checks to be able to ship on time and iterate over the implementation by using real world data.

No audit for policies targeting *

Certain policies can target any kind of Kubernetes resource. The safe annotations policy is an example of that.

Targeting all kind of resources would make more complicated to find all the Kubernetes resources contained inside of a Namespace that have to be audited.

To simplify things, we can state that policies targeting * are not going to be evaluated by the audit checks. Users have to be explicit about the Kubernetes resources a policy targets.

No live update of PolicyResult objects

The proposed workflows requires that only one process is actively auditing the resources of a given Namespace.

That's because:

• The PolicyResult of the Namespace is used to reduce the number of auditing evaluations performed
• Once all the policies and involved resources are audited, the PolicyReport audit is going to be overwritten with the results of the audit

Users can still get live data about the rejections/approvals by looking both at Prometheus data and Open Telemetry traces.

We should focus on making the process of creating a PolicyReport as fast as possible. The reports are recreated on a regular basis. If a user wants a fresh PolicyReport without having to wait for the next scheduled run, we can provide a way for the user to trigger this operation.

Finally, if we really have to implement live update of PolicyReport, we can come up with an alternative architecture that will allow for that. For example, we could have all the write operations of PolicyReport to go through a dedicated "writer" service.

[flavio]

Sorry for the long comment :sweat: . Let's talk more about that in a dedicated call instead of going through another round of comments. This is going to be faster

[raulcabello]

The objectSelector is of type metav1.LabelSelector same as the namespaceSelector. That means we can use the same method FormatLabelSelector to convert it to the type we need for doing the filtering.

This changes the implementation proposed since we have to apply the objectSelector to get all pods that are relevant for a specific policy in a namespace. This is what we can do in order to avoid doing extra calls to the api server:

• Create the dictionary proposed in step 2
• Check if the resource matches the objectSelector before doing an evaluation of a policy. This can be done using the Matches method which doesn't make any call to the apiServer.