confidential-containers / documentation

Documentation for the confidential containers project
Apache License 2.0
73 stars 48 forks source link

[RFC] Proposal for KBS Resource URL #85

Closed Xynnn007 closed 1 year ago

Xynnn007 commented 1 year ago

Background

Currently, CC Key Broker System (including CC-KBC and Generic KBS) is under development. At the meantime, we have started to use other Key Broker Systems (like Offline-FS-KBC, EAA-KBC & Verdictd to distribute some secret resources (key resources, like policy.json for image security, image signature verification key, image registry credential, etc.). However, we still need an identifier to uniquely identify a specific KBS resource, because without that mechinism, for example, we cannot request different image verification keys inside one Pod from KBS. Let's explain it.

Currently, the way we use different KBS, is to hard-code a key which is used to identifier the resource inside the code, like policy.json and cosign verification key in image-rs. Then, we prepare different resources for each hard-coded key in experimental KBSes, such as resources.json for offline-fs-kbc and cosign verification key, policy for verdictd. Here the description granularity of the key here is very rough, which means that if we want to request a public key, there will be no identifier to determine which public key is required, thus making it unable to support multiple public key in both image-rs and KBS side.

Similar problems also occur in Credentials of image Registry, image security policy files, etc.

Detailed Description of Current Implementation

Let's take the scenario where using image-rs to request the public key from KBS (Verdictd) to perform image signature verification as an example.

What we do now is to hard-code a constant "Cosign Key" inside image-rs. Every time image-rs needs to request KBS for a public key, it will do the following steps:

  1. Use constant "Cosign Key" to construct a ResourceDescription struct {"name": "Cosign Key"}
  2. Use that ResourceDescription as input parameter to request GetResource gRPC API of Attestation-Agent (AA for short in the following)
  3. AA uses EAA-KBC plugin to parse the ResourceDescription. Then it follows its own logic to access Verdictd on the other node to get the public key.
  4. AA then returns the key to image-rs.

image

The problem here is as mentioned:

Similiar resources includes credentials to access private registries, etc. If there are more and more resource types, and every resource has different instances on the KBS side, it seems impossible for us to use hard-code to meet this need.

To resolve this problem, we propose a KBS Uniform Resource Locator (URL) protocol. We can use KBS URL together with configuration file to support image-rs and other components to retrieve specific resource on the specific KBS server.

Let's quick go through how KBS URL can work with CC-KBC and generic-KBS.

About CC-KBC and generic-KBS

We hope that new KBS URL can work well with CC-KBC and generic-KBS. Let's take an example, still cosign public key. The diagram below shows each entity that carries the identity information of each call routine without KBS URL.

image

  1. First of all there should be a hard-coded Cosign Key field.
  2. image-rs uses this field to construct a ResourceDescription struct to request AA
  3. CC-KBC receives ResourceDescription and converts it to a web url due to KBS OpenAPI to access.

If we bring KBS URL and configuration file in, the workflow will be

image

  1. We can describe KBS resource in the configuration file (the link here is not KBS URL, only helps to explain) and image-rs reads it
  2. image-rs converts this KBS URL into a ResourceDescription
  3. image-rs uses this Description to request AA
  4. AA's CC-KBC plugin converts this ResourceDescription to a web url due to KBS OpenAPI
  5. CC-KBC access the web url to retrieve the KBS resource, s.t. public key.

Target

This proposal aims to

This is especially critical for many scenarios, like

KBS URL Protocol

There are two options to define KBS URL Protocol.

Option 1

This option uses path-like scheme to define KBS URL. The format is as following

<kbc-type>://<url-of-kbs>/<repository>/<type>/<tag>

It seems like a url for image, which has scheme <registry>/<repository>:<tag>. Here,

For example, Alice has an instance key1 of resource type Cosign-Key on KBS server example.cckbs.org. We can use cckbc://example.cckbs.org/alice/Cosign-Key/key1 to uniquely identify the key.

warning: here <repository>, <type> and <tag> only supports uppercase and lowercase letters, numbers, hyphens, and underscores.

Option 2

This option uses query-like scheme to define KBS URL. The format is as following

<kbc-type>://<url-of-kbs>/<key1>=<value1>&<key2>=<value2>&...

Here,

Here, the query conditions can be freely set by the user. For example:

I Prefer to Option 1, because it seems more tight, and somehow we may not need very flexible query machinism like option 2. Please share your opinions.

About KBC Types

Although there are some different KBC types, like cc-kbc, offline-fs-kbc, etc.

So, we can define <kbc-type> field like the following:

Here we distinguish different kbc's protocol because different kbc may handle key_id in different way. And we may go on a talk whether we agree on a uniform protocol kbs:// to cover all the mentioned kbcs.

Use cases

This section clearifies the producer and the consumer of KBS URL.

Producer

Here are two examples.

Configuration of Image Security policy.json

We can use KBS URL in the policy.json to specify a public key to verify the given image like the following (using Option 1 for example)

{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
            "quay.io/kata-containers/confidential-containers:cosign-signed-1": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "cckbc://example.cckbs.org/my/Cosign-key/1"
                }
            ],
            "quay.io/kata-containers/confidential-containers:cosign-signed-2": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "cckbc://example.cckbs.org/my/Cosign-key/2"
                }
            ],
        }
    }
}

We can use key "cckbc://example.cckbs.org/my/Cosign-key/1" for image quay.io/kata-containers/confidential-containers:cosign-signed-1 and key "cckbc://example.cckbs.org/my/Cosign-key/2" for image quay.io/kata-containers/confidential-containers:cosign-signed-2 when doing signature verification.

Annotation Packet Generated by CC-KBS When Encrypting an Image

In current CCv1 image security design, there is a entry with key "org.opencontainers.image.enc.keys.provider.attestation-agent" in the manifest of a encrypted image layer. The entry's value is called Annotation Packet. The Annotation Packet contains the encrypted LEK (Layer Encryption Key, the key to encrypt the layer) by a KEK (Key Encryption Key, the key to encrypt LEK), an id of the KEK of some KBS, like the following

{
    "key_id":"1234",
    "wrapped_data":#base64encode(Enc(PrivateLayerBlockCipherOptions)),
    "wrap_type":"",
}

Here "key_id" specifies the concrete KEK inside a given KBS. However, this way might only fit in experimental environment, because the owner of the image might have multiple KBS. We bring in KBS URL here similarly, s.t.

{
    "key_id":"cckbc://example.cckbs.org/my/key/1234",
    "wrapped_data":#base64encode(Enc(PrivateLayerBlockCipherOptions)),
    "wrap_type":"",
}

Now when decrypting the layer, it will be more conrete to know where the decryption key is.

Consumer

Image-rs

The KBS URL in the first example above is directly consumed by image-rs, following the steps:

  1. image-rs reads the KBS URL from the policy.json
  2. image-rs converts the KBS URL into a ResourceDescription in a determined way
  3. image-rs uses the ResourceDescription to request AA's GetResource gRPC API

CC-KBC

The KBS URL in the second example above is consumed by CC-KBC, following the steps:

  1. When an image layer needs to be decrypted, image-rs and ocicrypt-rs will extract the layer's Annotation Packet from the manifest of the image.
  2. ocicrypt-rs will call AA's UnWrapKey gRPC to send the Annotation Packet to AA.
  3. AA will parse Annotation Packet and get key_id field.
  4. Based on the <kbc-type>, AA will deliver the key_id to related KBC plug-in. In this case it will be cc-kbc, and CC-Kbc will convert the KBS URL into a OpenAPI Web URL of KBS, s.t. from cckbc://example.cckbs.org/my/key/1234 to something like https://example.cckbs.org/kbs/v0/key?repository=my&type=key&tag=1234 (This web url depends on how KBS OpenAPI defines)
  5. CC-KBC access the target web url to retrieve the decryption key

Behavior in Kata-CC

Fow now, only one KBS is used by a pod in Kata-cc and its address is delivered by a config file. So we can do some specific thing for kata-cc:

Influence

To KBS Protocol

Way 1: Without modifying OpenAPI definition

We can implement two functions:

Still, some suggestions for KBS protocol should be considered in Way 2

Way 2: Some modification for OpenAPI definition

KBS protocol is defined by OpenAPI. Currently, apis for resource retrievement are two:

We suggest to expend the definition, s.t. use resource to descript all kinds of resources including key, token, etc. How to process is implemented by the KBS server itself. In this way, we can have two strategies to modify the API design due to differernt choice of KBS URL option.

Path-like KBS URL

For KBS URL using option 1, s.t. <kbc-type>://<url-of-kbs>/<repository>/<type>/<tag>

The definition of resource OpenAPI can be something like

/resources/{repository}/{type}/{tag}:
  get:
    ...

And the url of HTTP GET request will be like https://example.cckbs.org/kbs/v0/resources/alice/Credential/database

Query-like KBS URL

For KBS URL using option 2, s.t. <kbc-type>://<url-of-kbs>/<key1>=<value1>&<key2>=<value2>&...

The definition of resource OpenAPI can be something like

/resources:
  get:
    parameters:
      - in: query
        name: resource
        required: true
        explode: true
        schema:
          type: object
        style: form

to support free query form. For example, the requested url can be

To image-rs

Signature Verification

Now, a policy.json for image-rs looks like

{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
            "quay.io/kata-containers/confidential-containers:cosign-signed": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/run/image-security/cosign/cosign.pub"
                }
            ],
        }
    }
}

It means that the public key is in path "/run/image-security/cosign/cosign.pub". If there is not, image-rs will request KBS for "Cosign Key". If we bring KBS URL in, arbitrary resource can be used. The policy can be

{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
            "quay.io/kata-containers/confidential-containers:cosign-signed": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "cckbc://example.cckbs.org/my/Cosign-key/1"
                }
            ],
        }
    }
}

Means that this image needs a public key tagged 1 in the repository named my of generic-KBS server example.cckbs.org.

The same mechinism applies to registry credentials, policy.json, etc. In future, more scenarios could use this url mechinism for not only image-rs.

References

https://github.com/apigee/registry/blob/main/openapi.yaml https://spec.openapis.org/oas/latest.html#paths-object https://github.com/confidential-containers/image-rs/issues/50 https://github.com/confidential-containers/guest-components/issues/218

Xynnn007 commented 1 year ago

This might affect KBC, generic-KBS, image-rs in a way. Please take some time and offer some comments. Thanks! @sameo @fitzthum @jialez0 @arronwy

jialez0 commented 1 year ago

@Xynnn007 Thanks! This is a very meaningful proposal. At present, in multiple scenarios on the node side, we use a variety of custom ways to index the resources we need from KBS, which has brought us a lot of confusion. The unified KBS URL protocol proposed in this proposal may can solve this confusion well.

For the options of the KBS URL protocol, I prefer the first option, which is to use the path like scheme to define the KBS URL. In fact, the second option does not uniformly describe the protocol for us, but only unifies the transmission format. The confusion caused by different form fields in different scenarios will still exist.

Currently, I have no good idea about how to cooperate with KBS OpenAPI. If our KBS resource OpenAPI also uses the path like scheme, it is good and clear, but I am not sure whether this will have some impact on the universality of the KBS protocol. @sameo Do you have any opinion on this?

sameo commented 1 year ago

@Xynnn007 Thanks a lot for putting such a detailed and thorough issue.

As @jialez0 I prefer the first option, and as a matter of fact the path-based scheme is already implemented.

I dont think we're going to need anything else from the KBS implementation (except from maybe defining an architecture and API to add new resources) and most of the remaining work is on the image-rs and AA/KBC side of things. This proposal (based on the first option) will decouple the KBC image creation process from its unwrapping, so it's definitely needed. It's a big +1 from me.

fitzthum commented 1 year ago

Great proposal. I just have a few comments.

First, once we adopt a universal resource id, it might seem like the unwrap_key and get_resource endpoints will do the same thing. From the KBS perspective this is true, but the API of the AA is different because ocicrypt-rs is expecting to use the keyprovider API. It's fine to have two different APIs for keys and other resources, but there are a couple things to think about. First, does this protocol support people who might want to use truly remote key unwrapping. As in, some people might not want the key that unwraps the annotation data to leave their KBS. Instead they might want to send the entire annotation to the KBS and get the unwrapped version in response. Would we be able to do that? On the other hand, it might be worth thinking about ditching the keyprovider API altogether. We've talked about unifying the container encryption scheme. One way to do this would be to move the unwrapping into ocicrypt-rs, which would only use the get_resource endpoint to request individual keys. This would be a pretty big change (and it wouldn't be compatible with the remote unwrapping thing I mentioned before), but it's something to think about.

Second, I am a little bit wary of having the KBS type in the URI. In general we want containers to have as broad compatibility as possible. I see that you are proposing having a generic KBS type as well, which could be useful. In some ways it might be better to specify which KBS is expected than to have silent failures. I'm not exactly sure what the right approach is yet. I think that we will be able to support this URI in the simple-kbs with very minimal changes although currently we have no concept of a repository, only a type and a tag. We could add this or we could just ignore the repository. The question of whether one KBS is compatible with another is kind of complicated. Like I said, I'm not sure the best solution, but I am wary of hardcoding the KBS into the image.

fitzthum commented 1 year ago

Another question is how the KBS should handle access control for resources. The KBS should probably have some mechanism for allowing certain guests to access certain resources (based on the result of the AS). Should this be individual resource level or more general (i.e. the repository level). For simple-kbs we allow every resource to have its own policy for when it should be released, but this is probably overkill and with multiple platforms the configuration space grows significantly. On the other hand we don't want every resource to be available to anyone with a valid attestation.

Xynnn007 commented 1 year ago

Hi @fitzthum , thanks for your deep thinking about the proposal! I'm very happy to share my views.

Surely unwrap_key and get_resource are doing some overlapping things, but I think it is reasonable to keep both because unwrap_key follows ocicrypt and might not expose the key (resource) out of KBS, while get_resource should expose the resource out of KBS and only defined in CoCo.

First, does this protocol support people who might want to use truly remote key unwrapping. As in, some people might not want the key that unwraps the annotation data to leave their KBS. Instead they might want to send the entire annotation to the KBS and get the unwrapped version in response. Would we be able to do that?

Yes, you're right and we can as we do currently. When we changed API from decrypt_payload to get_key, we found that KBSes using some kinds of KMS as backend did not support export the key, making decryption only happen inside KBS.

On the other hand, it might be worth thinking about ditching the keyprovider API altogether. We've talked about unifying the container encryption scheme. One way to do this would be to move the unwrapping into ocicrypt-rs, which would only use the get_resource endpoint to request individual keys. This would be a pretty big change (and it wouldn't be compatible with the remote unwrapping thing I mentioned before), but it's something to think about.

I may have some objections. The reasons are

Besides, as that image encryption in ocicrypt-rs, cosign image signing in sigstore-rs are prepared, we think it is time to provide a integration tool when generic KBS is prepared. The tool will encrypt (in ocicrypt way) & sign (in cosign way) the image and registry the KEK into KBS.

Second, I am a little bit wary of having the KBS type in the URI. In general we want containers to have as broad compatibility as possible. I see that you are proposing having a generic KBS type as well, which could be useful. In some ways it might be better to specify which KBS is expected than to have silent failures. I'm not exactly sure what the right approach is yet. I think that we will be able to support this URI in the simple-kbs with very minimal changes although currently we have no concept of a repository, only a type and a tag. We could add this or we could just ignore the repository. The question of whether one KBS is compatible with another is kind of complicated. Like I said, I'm not sure the best solution, but I am wary of hardcoding the KBS into the image.

I agree with you. Hardcoding the KBS into the image includes two aspects:

Another question is how the KBS should handle access control for resources. The KBS should probably have some mechanism for allowing certain guests to access certain resources (based on the result of the AS). Should this be individual resource level or more general (i.e. the repository level). For simple-kbs we allow every resource to have its own policy for when it should be released, but this is probably overkill and with multiple platforms the configuration space grows significantly. On the other hand we don't want every resource to be available to anyone with a valid attestation.

Exactly! Overall, I agree with you. I think (pod/user) identification and access control are another problem to handle, which is related to the identity of TEE/tenant/container/pod we can talk in another proposal/issue. There are some initial thinking I want to share:

Xynnn007 commented 1 year ago

Second, I am a little bit wary of having the KBS type in the URI. In general we want containers to have as broad compatibility as possible. I see that you are proposing having a generic KBS type as well, which could be useful. In some ways it might be better to specify which KBS is expected than to have silent failures. I'm not exactly sure what the right approach is yet. I think that we will be able to support this URI in the simple-kbs with very minimal changes although currently we have no concept of a repository, only a type and a tag. We could add this or we could just ignore the repository. The question of whether one KBS is compatible with another is kind of complicated. Like I said, I'm not sure the best solution, but I am wary of hardcoding the KBS into the image.

Now a kata-agent will be given the type of KBC, address of KBS. Then it will call inside image-rs for resources, however, only to one KBS specified by launch parameter of kata-agent. I think this can be better. As actually the kata-agent does not need to know which KBS or KBC the user will use. As for the problem of distribution of KBS public key, the pubkeys are distributed from the user. The user can give many public keys, which share the same logic of one key, even if some of them will not be used.

In this way, type of KBC and the address of KBS should be included in a URI where the resource is used, by which we can reduce another parameter in the startup parameters of kata-agent, and let the image-rs to flexibly support request resources from different KBS. There is another problem if we keep a universal kbs://, that the AA logic does not know which KBC should be dispatched to.

I do not get a good way to deal with this, up to now -

c3d commented 1 year ago

Hi @Xynnn007. Thank you for the detailed write up and rationale.

Overall, I like it, and I too have a preference for option 1. However, I am not convinced by the rationale for needing different kbc types. Isn't the resource type sufficient for the KBS to figure out if it's something it knows how to deal with? Normally the "schema" portion of a URL is intended to describe the type of protocol, and as long as you expect the underlying protocol to be the same, I think that you should use a single schema for simplicity. You suggested yourself using 'kbs' to cover all the KBCs, and I believe we should aim for that from the start, with a well-defined response when the KBS does not know how to deal with a given resource type.

Xynnn007 commented 1 year ago

Hi @c3d thank you very much for your comment. Please check the previous comment of your comment. To sum up, currently the KBS info is given from kata-agent, which I prefer to abondon to include this information in the uri of the resource. In this way we can support to request resources from different KBSs.

If we do not need this for AA, a universal kbs:// is good.

fitzthum commented 1 year ago

@Xynnn007

As I've mentioned before, I think getting resources from multiple KBSes should be thought of more as an anti-pattern than something we should support. There are some very tricky security issues that come up when we do this.

Even if we just have one KBS, it's hard to make sure that we are talking to the right one. Putting the public key of the KBS in the measurement does not solve this problem, because a malicious KBS could simply ignore this part of the measurement. I write about this at length here. Getting signatures and secrets from different KBSes is particularly dangerous as mentioned in https://github.com/confidential-containers/guest-components/issues/194.

You mention that we could give multiple KBS public keys to support multiple KBSes, but again this is not as reliable as it seems and even if it was, then each KBS would have to know the public keys of all the other KBSes that it trusted in order to validate the measurement.

larrydewey commented 1 year ago

Is it possible to have multiple KBS URIs specified in a configuration, and specify a priority for which URI should be used primarily?

c3d commented 1 year ago

@fitzthum wrote:

As I've mentioned before, I think getting resources from multiple KBSes should be thought of more as an anti-pattern than something we should support. There are some very tricky security issues that come up when we do this.

As discussed in today's meeting, there may be legitimate contexts where we actually need multiple KBS's, notably in teh case where a device like a vGPU needs its own secrets from the KBS. The consensus in today's meeting seems to have been that any encrypted communication between CPU and GPU could be managed with a Diffie-Hellman key exchange. However, I could think of things like license keys (NVIDIA has been known to require specific licenses to enable some features) that could come from a KBS.

So while I agree that having multiple KBSs being able to deliver the same secrets looks like something we should prevent as much as possible, having multiple KBSs that deliver secrets to different components might be a valid option. So I'm wondering if it's possible to have something in the URI that let's us distinguish between valid and invalid configs. Like having some sort of format convention for all URIs where we could say "if this or that part of the URI is the same, then we can safely reject".

dcmiddle commented 1 year ago

@hdxia please see if this would be compatible with your proposed KBS.

fitzthum commented 1 year ago

@c3d

So while I agree that having multiple KBSs being able to deliver the same secrets looks like something we should prevent as much as possible, having multiple KBSs that deliver secrets to different components might be a valid option. So I'm wondering if it's possible to have something in the URI that let's us distinguish between valid and invalid configs. Like having some sort of format convention for all URIs where we could say "if this or that part of the URI is the same, then we can safely reject".

It's not as simple as avoiding delivery of the same secrets. The most important thing is probably to make sure that signature validation information comes from the same source as any sensitive/secret material. In general we want to enforce that an enclave belongs to only one client. We can run into major problems if we start mixing secrets and policies from different parties.

I think it might be hard to enforce this at the level of the resource URI. We don't really control the names of resources. They could even use two different names to identify the same resource. I'm not sure how we would detect this.

Fortunately I think the GPU stuff is operating at a slightly different scope. If there is some component inside the guest that needs to connect to a license server as part of GPU attestation, I think that is fine. It doesn't need to interact with the KBS at all and shouldn't compromise any of our guarantees. My thinking is that as long as we are measuring whatever this component is, we can more or less let it be.