[RFC] Attestation Server

The Attestation Agent project is a Confidential Containers project that provides a modular an pluggable architecture for having CC guests being provisioned with keys and/or secrets from a key brokering server.

The attestation-agent architecture allows for different implementations to be added and built into the binary, through Key Brokering Client (KBC) implementations. This approach provides ways for adding CSP and hardware specific implementations to the client side of the attestation architecture of the Confidential Containers project.

Adding a KBC into the attestation-agent code base usually brings a protocol and hardware dependency on how this new KBC would communicate with the key brokering service (KBS). As a consequence, I think the Confidential Containers project should be more explicit and clearer on the assumptions, definitions and architecture of the server side of the attestation services.

So far, the Confidential Containers project scope has been limited to the node domain, i.e. to software and hardware components running on the cluster orchestration worker node. Defining the attestation services expectations and architecture, its scope would expand outside of the worker node, but I'd argue that given the strong dependencies of the project on that part of the overall confidential computing architecture, this expansion is natural and unavoidable.

How far we go into defining the attestation service architecture is a discussion I'd like to start through that issue. At that point, I think we could:

Document the existing KBS, Attestation Servers and Key Management Services and describe which ones are supported by our attestation agent implementation.
Define and implement an attestation-server project that would be the server side for attestation-agent KBCs. KBCs would call into the attestation-server endpoints. Similar to the attestation-agent project, we would provide a modular architecture for hardware and CSP specific implementations to be added to the code base as buildable plugins, in order to decouple the verification and key management parts from the hardware agnostic communication protocol between the KBCs and the attestation server. The attestation-server would call into verifier plugins and talk with key management services through specific plugins as well, in order to verify and retrieve secrets and keys from the overall KBS.

I think the first point by itself, although desirable, is not enough and we should work on both points. Working on the second point would be a forcing function for more clearly defining the API between our KBCs and the attestation backends. This is a personal opinion that, I hope, will trigger some constructive feedback from the attestation and KMS experts from this community.

References:

The kbs-rs project: https://github.com/confidential-containers/kbs-rs/
Pick a name issue for kbs-rs: https://github.com/confidential-containers/kbs-rs/issues/8

In the initial issue in the attestation agent, there's some definitions I'll try to be consistent with and try to reflect the understand I got from that discussion.

The Attestation Agent supports pluggable Key Broker Clients (KBC). Each KBC is meant to know how to interface with a specific Key Broker Server (KBS). Each KBS is responsible for checking the attestation and releasing a key back across the secure connection to the KBC. The verification and key retrieval might be integrated in the KBS or call separate services for those actions e.g. Intel Attestation Service and Azure Key Vault.

The interface, then, between the node and the remote key is between the KBC and the KBS. From a terminology perspective then I'd suggest we drop the Attestation Server term used in the initial post for this issue. I think that term, Attestation Server, would be more related to the component integrated in the KBS or called by the KBS to verify the attestation evidence.

As mentioned in the initial post this project is restricted to the node right now and the natural boundary then is KBC<-->KBS. We did not define a general API that a KBS would conform to but instead a plugin architecture requiring the KBC. I think that was necessitated by the variety of KBS implementations depicted in the AA issue.

I think it would be awkward to introduce KBS implementations into the Confidential Containers project directly. In addition to the variety, the mature implementations pre-exist our project and have use cases separate from ours. Imagine that a KBS might release a key for a variety of use cases and policies, not just what our container project focuses on.

At the moment we are using the terms Attestation Server and Key Broker Server more or less interchangeably. I have a hard time picking a favorite. On one hand I think that Attestation Server is a much better term. For people who do not know the project I think it is far more intuitive and the term is already used outside of the project to refer to the same thing. I am not familiar with KBS being used anywhere else and I think it is dangerously similar to KMS. I've seen that confuse a number of people. Furthermore KBS is a bit of a misnomer because it doesn't necessarily release keys or at the very least it doesn't exclusively release keys (consider the policy file for signatures). On the other hand an attestation is always involved.

On the other hand, we have basically the same point that Dan mentioned above. It makes sense that the thing the KBC talks to should be the KBS rather than the AA. Of course this is only useful for people who know that the AA has modules called KBCs. People who don't know this, it might be more intuitive to have an Attestation Agent and an Attestation Server.

So tbh I think it might not be the worst thing to just continue using the terms interchangeably, which is probably going to happen anyway even if we try to pick one.

It is better not to use KBS with Attestation server (AS) interchangeable. They are in different purpose and misuse them will cause confusion. RATS architecture already defines these roles. lets be consistent, otherwise, lot people may scratch their heads.

KBS - key broker service, it is a front-end proxy/broker for its client to get keys without worrying about different key management protocols. KBS may support different key management services (KMS) and there are tons of them. KBS may support key policies to decide whether it needs to release the key per client request. these key policies will trigger attestation of the client (in this context). of course, you can have policy not related to attestation. If the policy says to attest/verify the client, attestation happens. Where the attestation happens is implementation dependent. Some KBS implementation (such as Ali verdictd has attestation/verifier included). Other KBS (ISecL) calls into a dedicated external attestation server/verifier to do the verification. Per RATs architecture, KBS is a relying party. and the current implementation of Kata CC with the KBC on the node is background model.

Attestation server/attestation service, is the verifier in RATs architecture. What it does is to verify the evidence (SGX/TDX quote, or SEV related evidence) and makes a claim whether these evidence match user defined policies. Then it issues attestation result/token. Attestation server is not in business of key management, which should be the job of relying party per RATs architecture.

Same concern here:) Attestation servers/KBSs are almost shared infrastructure services, which may not be dedicated for Confidential Containers only. And there are multiple pre-existing vendors/service providers, so what's the benefit to include kbs implementations in CC project? The answer may be that we need a sample kbs implementation for casual users to try CC:)

If CC needs a general KBS/AS, it must implement a dedicated KBC to support it. It is unrealistic that adapting existing KBS/AS to the KBC corresponding to the general KBS/AS for CC. Instead, implementing various vendor specific KBCs for existing KBS are more realistic. Therefore, the general KBS/AS still keep the rule of 1:1 mapping between KBC instance and its corresponding KBS instance.

According to RATS spec, the interaction model and message format are already defined for the communication between attester/KBC and relying party/KBS. We just need to implement a communication protocol between them to cover the beed from AA (decryption key, policy, any personalized sensitive information and so on). As I mentioned in another thread, it is possible to use this protocol to address pre-attestation. Therefore, general communication protocol is part of general KBS.

@sameo About the design of general KBS/AS, as mentioned by you, there are several places to be modularization, including KMS, DB, appraisal policy engine for checking attestation results and so on. I think the interface to verifier/attestation verification service can be not modularization. Actually we need to follow related RATS spec to re-use the design experience of communication protocol between KBC and KBS. As @hdxia mentioned, attestation verification service/verifier may be a centralized authoritative service deployed in remote peer. The trust model between KBS and attestation verification service/verifier is different, which is the responsibility of relying party establishing the trust to attestation verification service/verifier. A verifier is responsible to directly verify the singed evidence, or indirectly through requesting the back-end attestation verification service provided by silicon vendor, and apply the evidence to appraisal policy. As a result, the policy engine returns a hardware irrelevant attestation results to KBS for further determination on whether authorizes the request initiated by KBC (e,g, a requesting secret). So CC also needs to consider having a reference verifier implementation as KBS to support possible TEE platforms. The modularization occurs at the so-called verification drivers in verifier which can verify various attestation evidences.

Perhaps along the same lines of this, I would propose that it may be easier to abstract out a generic implementation of a KBS, it was talked about during today's meeting as a dummy kbs, but I think it would be very interesting to see a set of generic or accepted unified set of architecture agnostic endpoints, which architecturally specific KBS implementations can then implement or consume. I am envisioning this as sort of a RESTful service which would be able to communicate with any of these KBS/AS instances.. I am currently working on a workflow of what this might look like, but I would love to hear what the community thinks about introducing something like this.

Edit: Reworded

Why does CC need a general KBS implementation? Before answering this question, it is necessary for us to understand why CC initially adopted the multiple KBC plugins mechanism.

I think the reason is the problem of "timing". The initial CCv0 architecture needs to quickly prove and show its potential and technical proposition, so everyone is busy building a functional prototype, that is, completing a minimum function set and drawing a function boundary. These are temporary and set for CCv0 based on the limited resources at that time. However, CC architecture itself needs to move towards production, which is the current goal of CCv1. We have seen that the function of CC has been expanded compared with CCv0, so its function boundary naturally needs to be extended as well.

Let's look at how we design and implement the attestation subsystem (including the entities implementing Attester, Relying Party and Verifier) in CCv0. First of all, attestation itself is very complex, so it is unrealistic to build a general KBS for CC at the beginning. In addition, there were already some KBS and ones born to support CC at that time. Therefore, we need to design a modular KBC mechanism that can access different KBS to support all possible KBS. Thanks to the flexibility provided by this mechanism. CC community has successfully let experts in the field of attestation do what they are good at. At the same time, many people from other projects and company can focus on the necessary feature set for CCv0.

So far, people have discussed a lot about CC boundary, and how to take into account the different KBS implementations from vendors, etc., but I think some of us miss one point: that is, KBS itself is actually a part of a specific attestation subsystem solution. It is just an specific instance derived from the upper level definition of the role "Relying Party" in RATS under some scenarios. In other words, KBS is not a term defined by the RATS architecture. It is just a concrete implementation of the RAT role "Relying Party". In fact, CC itself is also a specific application of confidential computing in cloud native. Therefore, as RATS architecture defines a complete attestation model, CC naturally needs a specific and complete attestation subsystem solution. As we can see in the RATS architecture, such a solution should adopt a general model based on the Background Check Model with three basic roles: Attester, Relying Party and Verifier. It can prove that the minimum function set of CC's attestation subsystem is complete and secure. Therefore, rather than simply adding a general KBS to CC, essentially we are talking about how to complete CC's attestation subsystem. Right now CC only has an attestation-agent acting as "Attester" role.

I think it is the best time to discuss a general KBS now. Gather experts in the field of attestation and start to seriously consider how to implement a general KBS for reference in CC. The positioning of this general KBS should only have a minimum set of functions, comply with RATS architecture and relevant standards, provide scalability for normal users who cannot develop a proprietary KBS system, and most importantly, prove its completeness and security.

There is nothing better than having a group of professional people gather in the same community to create an core subsystem, especially when this subsystem is related to security. Many people think that Kata CC is an extension of Kata, which is not wrong, but I think our essential work is to let Kata accommodate the new trust model and harden the security at the boundary in TEE side. From this point of view, CC needs to consider the security of KBS as a part of its overall security (we all know the classic barrel principle). General KBS must set an example, providing a reference implementation that fully conforms to the threat/trust model CC defines, with a high security baseline for normal users.

I agree with Sameo that both options are to be desired. The engineering essences behind these two options are: write a new KBC (or properly modify the existing KBS) to support for CC use case for existing KBS, or write a general KBS for reference. Implementing a general KBS for CC will not reduce the competitiveness of other existing KBS, because CC still retains the ability to connect with existing KBS by adding new KBC plug-ins. At the same time, the general KBS is also a highly modular systems, like containerd, and it even can't do anything without the supports offered by various plug-ins. But this is the real competitiveness of general KBS lies in this: the community is only responsible for building a general KBS for reference with the minimum function set and security features required by CC. It is not to replace the existing KBS, but to help normal users more easily to bridge CC's TEE part. In this way, the normal users can take more time on how to connect the existing business back-end to the general KBS through standard interface/protocol. This positioning of a general KBS has great attraction to people without the relevant technical background of attestation. At the same time, it also avoids the risk of building a KBS from scratch with potential security vulnerability, high coupling between components, and even incomplete by the normal users who have to build a KBS from scratch in order to avoid the demands on specific KBS vendors and proprietary KBS. Actually the competitiveness of vendor-specific and proprietary KBS is also here: KBS vendors can still use their favorite, more powerful, more integrated and more competitive approaches to implement their own KBS interaction protocol and KBS back-end. CC community does not kill KBS vendors. It still reserves the extension of interacting with various KBS from different vendors.

Should we try to make a universal KBS? I think it is at the very least worth considering.

First of all, the current state of our attestation infrastructure is not bad. It was a good choice to make the AA modular. No matter what happens people will always be able to make add their own KBCs that support whatever KBS they want to use. There are a few existing KBS options. That said, I don't think they are quite complete. To echo what @jiazhang0 said above, most of the existing implementations know how to check if a launch measurement is valid, but they don't really know how to enforce policies.

In some ways checking that the measurement is valid is the easy part. Policy enforcement is where things get interesting. By providing keys and signature policy files, the KBS determines the security parameters of the enclave, essentially serving as a trusted control plane. In the simplest case a KBS can just check a launch measurement against a list of allowed measurements and release whatever keys the AA requests. Imagine another example where a guest owner has some workloads that are more sensitive than others. They may want to make sure that for their most sensitive workloads additional endpoints of the Kata Agent API are disabled. Since locking down the endpoints changes the guest launch measurement, the KBS can enforce this policy. Maybe the guest owner also wants to run certain containers only at certain times of the day. Maybe they want to make sure that particular containers never run together in the same pod. Essentially we might want to use the KBS as a trusted admission controller. Some of these policies are more easy to guarantee than others and we probably wouldn't support any of them in the first step. Currently it's difficult to enforce any policy at all and I think that is a big issue.

Finally, let's be honest. There are only 3 KBCs in the AA today. offline_fs_kbc is for platforms that have encrypted boot images (currently only Secure Execution) or non-secure demos. offline_sev_kbc works with the simple_kbs, which attempts to answer some of these policy questions but is only scratching the surface and is only for pre-attestation. eaa_kbc is made for verdictd whose architects seem to be in favor of a universal KBS. These are good options, but I think we could do a lot better.

Ok, I guess we can close this given that we are now well on our way to having a generic KBS and AS.

confidential-containers / documentation

[RFC] Attestation Server #26