New KBS endpoint: `/register`

[tylerfanelli]

Currently, the KBS protocol offers a standard interface for clients to attest their TEEs. Multiple implementations of a KBS server could be created, but as long as the protocol is followed, KBS clients should be able to attest their TEEs without caring about which server implementation is being used.

One crucial part of attesting a TEE is certifying its launch measurement. Certifying a TEE's launch measurement is done by

1. Registering an expected launch measurement with the KBS server before the TEE is attested.
2. Fetching the attestation evidence from the KBS client and finding the actual launch measurement from its respective attestation report.
3. Comparing these two launch measurements to ensure that they match. This ensures that the TCB of the TEE is certified to be measured correctly.

An example of this using SHA512 measurements:

┌─────────────────────────────┐             ?            ┌───────────────────────────┐
│ Expected launch measurement │           ┌───┐          │ Actual launch measurement │
│                             ├──────────►│ = │◄─────────┤                           │
│          SHA512             │           └───┘          │          SHA512           │
└─────────────────────────────┘                          └───────────────────────────┘

Registering TEEs

Although there is a standard interface for clients to attest their TEEs, there is no standard interface for users to register their reference values with the KBS server for eventual use by a KBS client. Since this is crucial step in attestation, KBS may greatly benefit from a standard interface to register reference information. Although CoCo KBS currently provides the RVPS, this is not standard and is tailored for CoCo KBS. What I propose is an extension to the KBS protocol to standardize reference value registration.

This comes in the form of a new HTTP endpoint to KBS: /kbs/v0/register. /register would be a POST request with a new KBS type, Registration, given as input.

struct Registration {
    reference_measurement: String

    etc...
}

/register would then generate a UUID to be used as a key for storing the reference value measurements in KBS storage. /register would then send this UUID back to the user that has registered their TEE. A diagram showing this process is below:

┌──────┐                                 ┌─────┐            ┌─────────────────────┐
│ Host │                                 │ KBS │            │ KBS Reference Value │
└──┬───┘                                 └──┬──┘            │      Storage        │
   │                                        │               └─────────────────────┘
   │       ┌────────────────────────┐       │                         ▲
   │       │  POST /kbs/v0/register │       │                         │
   ├──────►├────────────────────────┼──────►│       ┌─────────────────┴──────────────────────┐
   │       │   Header: No Cookie    │       │       │            Generate UUID               │
   │       │   Body: Registration   │       ├──────►│ Parse reference values in Registration │
   │       └────────────────────────┘       │       │Store reference values with UUID as key │
   │                                        │       └────────────────────────────────────────┘
   │       ┌────────────────────────┐       │
   │       │        Response        │       │
   │◄──────┼────────────────────────┤◄──────┤
   │       │    Header: No Cookie   │       │
   │       │       Body: UUID       │       │
   │       └────────────────────────┘       │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   │                                        │
   ▼                                        ▼

With the UUID acting as a key, reference values are organized in KBS storage and can be fetched using the key (UUID) whenever they are needed to attest.

Client Usage

With the addition of a Registration UUID, a KBS client would need to:

1. Store the Registration UUID
2. Present the Registration UUID to the KBS server during the Attestation phase of protocol. The server would use the UUID to fetch the reference values.

Note that KBS clients do not register reference values (and thus do not inherently have access to the Registration UUID), but must still present them to the server somehow.

Each TEE offers an API used by hypervisors to make some data from the host made available in the attestation report. SEV-SNP names this field HOST_DATA. It is documented in Table 22 of the SEV-SNP ABI document. HOST_DATA is 32-bytes in size. Although I will use SEV-SNP as an example, fields with similar purposes are found in each TEE processor architecture.

What I propose is to use the HOST_DATA field to store the Registration UUID. Hypervisors could be supplied previously-fetched Registration UUIDs by input when booting confidential VMs, and could then embed these UUIDs in the HOST_DATA field of their guest's attestation report. HOST_DATA is unique to each confidential guest, and must be supplied at VM launch.

A diagram showing a prospective HOST_DATA is shown below:

       Attestation Report HOST_DATA
 ┌──────────────────┬───────────────────┐
 │       etc...     │ Registration UUID │
 └──────────────────┴───────────────────┘
31                16 15                 0

Once the UUID is embedded in the attestation report, it can then be presented to the KBS server as a part of the attestation report. This would require no changes to the KBS Attestation phase.

Using the Registration UUID to fetch Reference Values

During the Attestation phase of the KBS protocol, the KBS client presents its evidence in the form of an attestation report. Like previously mentioned, a crucial part of certifying the launch measurement of the report is to compare with a reference launch measurement.

At this point, the KBS server is already storing the reference launch measurement. With the attestation report now present, it can now fetch the reference values that were previously registered and compare. To do this, it must use the UUID embedded in the attestation report as a key to the reference values for the TEE. Using this key, the attester can fetch reference launch measurements, and do this comparison. A diagram of this process is shown below:

                 ┌───────────────────────────────────────────────────────┐
                 │         Fetching reference values using the           │
                 │  Registration UUID embedded in the attestation report │
                 └───────────────────────────────────────────────────────┘

┌───────────┐                                                          ┌───────────────┐
│  Attester │                                                          │ KBS Reference ├────┐
└─────┬─────┘                                                          │ Value Storage │    │
   ▲  │                                                                └───────────────┘    │
   │  │                                                                        ▲            │
   │  │                                                                        │            │
   │  │                                                                        │            │
   │  │                                                                  ┌─────┴────┐       │
   │  │                                                                  │   UUID   │       │
   │  │                                                                  └──────────┘       │
   │  │                                                                        ▲            │
   │  │                             Attestation Report                         │            │
   │  │                           ┌────────────────────┐                       │            │
   │  │                           │        ...         │                       │            │
   │  │                           ├────────────────────┤                       │            │
   │  │                           │        ...         │                       │            │
   │  │                           ├────────────────────┤                       │            │
   │  │                           │        ...         │                       │            │
   │  └────────────────────────►  ├────────────────────┤   ┌───────┬───────────┴──────┐     │
   │                              │        ...         │   │ etc...│Registration UUID │     │
   │                              ├────────────────────┤   └───────┴──────────────────┘     │
   │                              │        ...         │  31          ▲               0     │
   │                              ├────────────────────┤              │                     │
   │                              │     HOST_DATA      ├──────────────┘                     │
   │                              ├────────────────────┤                                    │
   │                              │        ...         │                                    │
   │                              └────────────────────┘                                    │
   │                                                                                        │
   │                                                                                        │
   │                                                                                        │
   │                                                                                        │
   │                                                                                        │
   │                           ┌───────────────────────────┐                                │
   │                           │      Reference values     │                                │
   └───────────────────────────┤  (For example, reference  │◄───────────────────────────────┘
                               │    launch measurement)    │
                               └───────────────────────────┘

Conclusion

I reiterate that although the KBS protocol provides a standard interface for clients to attest their TEEs, it does not offer a standard interface to register reference attestation evidence to a server in order to compare with. This proposed KBS step aims to provide a standard interface to register reference values without modifying any other KBS steps.

I'm hoping to gain a bit of perspective from the protocol designers regarding if they feel this would be beneficial to KBS. It is something that I think is very much needed.

Thanks!

@Xynnn007 @fitzthum @sameo @jialez0

[fitzthum]

This is an interesting proposal. I need to think about it a bit more, but here are a few questions.

First, as you mention we already have some support for uploading reference values in our AS/RVPS. Since our attestation service evaluates evidence flexibly using OPA, we allow a client (the person who owns the guest, not the KBS client) to upload not just one reference measurement but any amount of reference values. The client also uploads the OPA itself. Basically we could be dealing with quite a few artifacts. This proposal seems to be more focused on just one reference measurement per boot. For instance you show the host data holding just one UUID. Do you think this proposal can be extended to support keeping track of many reference values per guest?

Maybe a more fundamental question is whether this registration endpoint would need to be privileged. From your diagram it looks like it is not. For the AS/RVPS uploading a reference value is a privileged operation and can only be performed by certain people. The idea is that we want to be sure that the reference values to be the gold standard that we use to evaluate hw evidence. In your proposal it seems like the host can register any reference value they want and then boot a guest however they like as long as they populate the host data field correctly. This seems a bit fishy to me.

In practice, I think we're going to have a number of different uses for the HostData field and it's probably not realistic to plan to use even half of it for any particular protocol. We are probably going to need to create a separate HostData protocol that allows us to multiplex this field between a number of different protocols. Either way, I'm not actually sure that you need to measure the UUIDs at all. I guess it would allow someone to examine the evidence after the fact and see which reference values were used (as long as they also had access to mappings of UUID to value). Other than that, I don't see the purpose. Regardless of whether the UUID is measured, the KBS client must tell the KBS the correct UUID or the attestation won't work. Why use the host data for this, particularly when it has such limited bandwidth?

I think this proposal is striking at something that I still haven't really wrapped my head around, which is how we want to keep track of the identity of guests. Maybe we can discuss this at LPC.

[tylerfanelli]

This is an interesting proposal. I need to think about it a bit more, but here are a few questions.

First, as you mention we already have some support for uploading reference values in our AS/RVPS. Since our attestation service evaluates evidence flexibly using OPA, we allow a client (the person who owns the guest, not the KBS client) to upload not just one reference measurement but any amount of reference values. The client also uploads the OPA itself. Basically we could be dealing with quite a few artifacts. This proposal seems to be more focused on just one reference measurement per boot. For instance you show the host data holding just one UUID. Do you think this proposal can be extended to support keeping track of many reference values per guest?

Yes. I used the launch measurement as an example, but the proposed Registration KBS type that would be input to /register can contain any data that would be used for client attestation. Like so:

struct Registration {
reference_measurement: String,
policy_agent: String
.. other registration data ..
}

We would have to come to a consensus on what is needed in Registration. For instance, in a typical RVPS registration of a guest, what values are registered?

Because the there is a 1:1 mapping of UUIDs to guests, we can organize our KBS storage such that each UUID maps to one of the registration values, like so:

                                                         ┌───────────────────┐    ┌────┬────────────────────────┐
                                                         │ Registered launch ├───►│UUID│ registered measurement │
                                                   ┌────►│   measurements    │    └────┴────────────────────────┘
                                                   │     └───────────────────┘
                          ┌───────────────┐        │
                          │               ├────────┘
┌───────────────────┐     │ KBS Reference │              ┌───────────────────┐    ┌────┬────────────────────────┐
│ Registration UUID ├───► │ Value Storage ├─────────────►│ Registered policy ├───►│UUID│         OPA            │
└───────────────────┘     │               │              │      agents       │    └────┴────────────────────────┘
                          └───────────────┴────────┐     └───────────────────┘
                                                   │
                                                   │
                                                   │     ┌────────────────────┐   ┌────┬────────────────────────┐
                                                   └────►│ Other registration ├──►│UUID│        etc...          │
                                                         │        data        │   └────┴────────────────────────┘
                                                         └────────────────────┘

This is how I think this proposal can be extended to support keeping track of many reference values per guest.

Maybe a more fundamental question is whether this registration endpoint would need to be privileged. From your diagram it looks like it is not. For the AS/RVPS uploading a reference value is a privileged operation and can only be performed by certain people. The idea is that we want to be sure that the reference values to be the gold standard that we use to evaluate hw evidence. In your proposal it seems like the host can register any reference value they want and then boot a guest however they like as long as they populate the host data field correctly. This seems a bit fishy to me.

Can you elaborate on what you mean by privileged? As you mentioned, the reference values are the gold standard and we don't want the host running the TEE to have any access to them. As such, the client (not the host) is responsible for registering the TEE. How does RVPS handle privileges in registration?

For example, consider a client looking to attest a TEE on a potentially malicious host. The client can do the following:

1. Register the TEE with its respective reference values with the KBS server, receive the UUID from the server.
2. Launch the TEE on the host, giving the UUID and address of the KBS server as input.

The host never receives any of the registered reference values, and can now embed the UUID in the attestation report of the guest. The guest can then attest on its own using the UUID from the attestation report.

In practice, I think we're going to have a number of different uses for the HostData field and it's probably not realistic to plan to use even half of it for any particular protocol. We are probably going to need to create a separate HostData protocol that allows us to multiplex this field between a number of different protocols. Either way, I'm not actually sure that you need to measure the UUIDs at all. I guess it would allow someone to examine the evidence after the fact and see which reference values were used (as long as they also had access to mappings of UUID to value). Other than that, I don't see the purpose. Regardless of whether the UUID is measured, the KBS client must tell the KBS the correct UUID or the attestation won't work. Why use the host data for this, particularly when it has such limited bandwidth?

Bare-metal TEEs need to have some way of getting access to the UUID. In a lot of TEEs, attestation happens at a point before the OS boots. With that, there is no access to any shared file or memory buffer from the host that the guest could use to read the UUID it requires to attest (and even if there was, this could potentially compromise the confidentiality of the guest). With embedding the UUID in the report, the guests have a way to send them to the attestation server.

I think this proposal is striking at something that I still haven't really wrapped my head around, which is how we want to keep track of the identity of guests.

Exactly. I feel this is very much needed in KBS at the moment.

Maybe we can discuss this at LPC.

Yes, I agree. Let's do that.

[Xynnn007]

Overall, I think the api /register of such semantics would be of great use to workload identity. Also, leveraging HOSTDATA field to identify workloads sounds good to me. Before this, if we could resolve the following problems, the whole solution for workload identity would be clear:

1. As @fitzthum said, how we can multiplex HOSTDATA field.
2. The privilege to access the /register endpoint
3. Whether we need the reference measurement per boot

For Q3. IMO, probably yes. Attestation is a lower level semantics than authentication. CoCo-AS could help to verify the basic TCB (like measurement for TDX; MRTD, RTMR for TDX), but it is hard for AS to know what HOSTDATA mean. After CoCo-AS checks the basic TCB, KBS would check the HOSTDATA to see which pod is now authenticating and do some other checks for the attestation token responsed by CoCo-AS against an extra policy inside KBS if any rule is defined.

For Q1, I'm going to write an RFC named initdata injection mechanism.

BTW, based on initdata injection, we team have designed an overall solution of workload identity for TDX/SNP/CCA which I'm also going to propose. The endpoint /register could be of great use in the solution.

It includes

• How to define the Ids & privileges before launching the pods
• How to verify the identities and how to authorize them
• How to use the identies to access resources

My personal idea is that how to multiplex HOSTDATA is very important and has reached consensus. We might as well start from this point.

[tylerfanelli]

Overall, I think the api /register of such semantics would be of great use to workload identity. Also, leveraging HOSTDATA field to identify workloads sounds good to me. Before this, if we could resolve the following problems, the whole solution for workload identity would be clear:

1. As @fitzthum said, how we can multiplex HOSTDATA field.

For my reference, what else are we thinking can be embedded in HOST_DATA, such that it needs to be multiplexed?

2. The privilege to access the /register endpoint

See my comment to Tobin above. RVPS's handling of privileges could be used as reference here.

3. Whether we need the reference measurement per boot

I think yes, because the launch measurement could change per boot. If the measurement changes, this could be violating confidentiality.

For Q3. IMO, probably yes. Attestation is a lower level semantics than authentication. CoCo-AS could help to verify the basic TCB (like measurement for TDX; MRTD, RTMR for TDX), but it is hard for AS to know what HOSTDATA mean. After CoCo-AS checks the basic TCB, KBS would check the HOSTDATA to see which pod is now authenticating and do some other checks for the attestation token responsed by CoCo-AS against an extra policy inside KBS if any rule is defined.

AS is responsible for the attestation report certifying, correct? With that, it would need access to the attestation report and reference values. It already has an interface to access the report. It can now use that report to fetch the reference values. HOST_DATA would be TEE-architecture specific, although each of them offer such a field in their attestation reports.

[Xynnn007]

The initdata proposal https://github.com/confidential-containers/confidential-containers/issues/171

[jepio]

Bare-metal TEEs need to have some way of getting access to the UUID. In a lot of TEEs, attestation happens at a point before the OS boots. With that, there is no access to any shared file or memory buffer from the host that the guest could use to read the UUID it requires to attest (and even if there was, this could potentially compromise the confidentiality of the guest). With embedding the UUID in the report, the guests have a way to send them to the attestation server.

Can you share more about which bare-metal TEE we're talking about? There are many ways to pass these things: ACPI tables, (unmeasured) memory pages, nvram variables, a well defined section of the binary, fw_cfg, smbios. From this sentence I gather that the KBS server url must also be passed into the guest: "Launch the TEE on the host, giving the UUID and address of the KBS server as input."

I agree with Tobin that there is no need for UUID to be measured/passed through host data. It is essentially a hint to the remote side which appraisal policy to apply. If you really feel the need to include it in the attestation report you can pass it to the KBS through REPORTDATA. HOSTDATA is such a rare resource that we really need to think hard about usage, and it's usage should be for things that must be attested, not as a host-guest data interface.

As I'm writing this, and having just reviewed https://github.com/confidential-containers/kbs/pull/240 I feel I have to ask. Could you accomplish what you're describing through an extension of the AS protocol proposed by @Xynnn007: if a user can register an appraisal policy with the attestation service, that policy can contain the measurement values they want to validate, and will be receive a uuid for the policy. That uuid is later passed to AS to specify the policy to be attested against. Does that cover that use case?

An alternative way of thinking about "It is essentially a hint to the remote side which appraisal policy to apply" is what we do in AKS CoCo: the expected measurements are signed and the attestation client has access to the certificate (injected through a volume or env variable - other mechanisms could work for your context) and forwards it to the attestation service. The attestation service checks measurements against the certificate and the certificate against an expected root. This can be expressed using an appraisal policy as well.

[tylerfanelli]

@jepio @fitzthum @Xynnn007

I'm thinking of SVSM, which attests before the OS is booted and AFAIK doesn't have any interfaces for shared memory currently. I chose HOST_DATA because it would require no changes on the guest side at the moment, and I haven't seen HOST_DATA used for any meaningful purpose as of yet. From this conversation, I see that there will be a use-case for HOST_DATA. However, the crux of this problem is client identification, so let's ignore HOST_DATA for the moment.

Here is an example of potential KBS usage that is not within the Kata/Confidential Containers use case:

We're running two SEV-SNP VMs with QEMU/KVM and SVSM. In order to run, they need to unlock a LUKS-encrypted disk with a passphrase that they need to attest to retrieve. Each VM also has a different launch measurement and LUKS passphrase. You register the reference values of each VM like so (in pseudo-JSON):

Register {
    launch_measurement: "vm_1_expected_launch_measurement",
    secret: "vm_1_luks_passphrase",
}

Register {
    launch_measurement: "vm_2_expected_launch_measurement,
    secret: "vm_2_luks_passphrase",
}

Now, VM-1 would like to attest, it sends a kbs::Request to the attestation server:

Request {
    "version": "0.1.0",
    "tee": "snp",
    "extra-params": "",
}

The server stores the guest's KBS state so far, and returns a kbs::Challenge:

Challenge {
    "nonce": "$NONCE",
    "extra-params": "",
}

The VM now fetches its (SEV-SNP) attestation report, creates a TEE public key, and sends them to the KBS server via kbs::Attestation:

Attestation {
    "tee-pubkey": "$RSA-PUBLIC_KEY",
    "tee-evidence": "$SNP-ATTESTATION-REPORT",
}

Now, the attestation service must do the following:

1) Verify that the report is legitimately from a SEV-SNP PSP by verifying the report's signature with the CA-VCEK certificate chain (via AMD's KDS).

2) Compare the attestation report's MEASUREMENT field with the launch measurement that was previously registered for this VM guest.

For point 2: The attestation service knows that it is storing a corresponding reference launch measurement to compare with the report's MEASUREMENT field. But the question is: how does it know which reference measurement to choose, out of the two it is storing? There is no unique identifier to match reference values with the TEE guests in which they are attesting.

To expand a bit further: What if the two VMs in my previous example have the same launch measurement, but different LUKS passphrases? When one of the VMs successfully attests, and then fetches it's passphrase from the resource endpoint, how does the attestation service know which passphrase to send to the guest?

That is what the crux of this issue is. The UUID I'm mentioning is the unique identifier needed to match reference values with the TEE guests in which they attest. Does this need to come from HOST_DATA? No, but it's a convenient area to store it (i.e. in the attestation report).

It really makes no difference whether we are registering appraisal policies, launch measurements, secrets, or anything else. The question is, when multiple of these values are registered, how do we uniquely identify them?

[fitzthum]

I am hesitant about using the host_data field only as a configuration interface. While it might be convenient to do this, I don't think it scales very well. It's true that we get the host_data field basically for free. It is handy that it is exposed to the host, guest, and the verifier. Realistically, though, any mature workload is going to have some other configuration interface. Even something as minimal as the SVSM, for instance, will need to have some way to load a state blob.

The more I think about it, the more it seems like maybe provisioning/registering is out of scope for the KBS protocol. In the CoCo KBS/AS we actually have two levels of policy. One for the AS,, which defines how to verify hw evidence, and one for the KBS, which determines which secrets can be released and when. I think this shows that provisioning is highly implementation-specific. When it comes to provisioning it isn't really a question of communication between guest<->kbs anymore, which I think also hints that the KBS protocol might not be the right place. That said, this doesn't mean that the sort of flows you are proposing don't make sense.

[jepio]

@jepio @fitzthum @Xynnn007

For point 2: The attestation service knows that it is storing a corresponding reference launch measurement to compare with the report's MEASUREMENT field. But the question is: how does it know which reference measurement to choose, out of the two it is storing? There is no unique identifier to match reference values with the TEE guests in which they are attesting.

When you register (measurement, passphrase) with KBS, you're attaching a release policy to the key object that is created in KBS. You need to authenticate to KBS to interact with it (might not fully be there yet). The identifier of the key that was created is tied to the LUKS container in some way.

Existing TPM-based LUKS key release would actually create an encryption key in KBS, encrypt the LUKS passphrase and store the encrypted passphrase in the LUKS header. This flow solves your other problems too: even if you'd get another VMs KBS key (which would be bad) you wouldn't have access to their disk or their encrypted passphrase.

To expand a bit further: What if the two VMs in my previous example have the same launch measurement, but different LUKS passphrases? When one of the VMs successfully attests, and then fetches it's passphrase from the resource endpoint, how does the attestation service know which passphrase to send to the guest?

Authentication - an authenticated (machine) user will only have access to their own keys.

It really makes no difference whether we are registering appraisal policies, launch measurements, secrets, or anything else. The question is, when multiple of these values are registered, how do we uniquely identify them?

This doesn't directly answer your question but: there are appraisal policies in AS (multiple) and release policies on (every) key object in KBS. RVPS is attached to AS and is queried before the AS policy is applied.

In your use case you should be thinking in terms of release policies on key objects and not reference values in AS.