Closed Xynnn007 closed 1 year ago
This might affect KBC, generic-KBS, image-rs in a way. Please take some time and offer some comments. Thanks! @sameo @fitzthum @jialez0 @arronwy
@Xynnn007 Thanks! This is a very meaningful proposal. At present, in multiple scenarios on the node side, we use a variety of custom ways to index the resources we need from KBS, which has brought us a lot of confusion. The unified KBS URL protocol proposed in this proposal may can solve this confusion well.
For the options of the KBS URL protocol, I prefer the first option, which is to use the path like scheme to define the KBS URL. In fact, the second option does not uniformly describe the protocol for us, but only unifies the transmission format. The confusion caused by different form fields in different scenarios will still exist.
Currently, I have no good idea about how to cooperate with KBS OpenAPI. If our KBS resource OpenAPI also uses the path like scheme, it is good and clear, but I am not sure whether this will have some impact on the universality of the KBS protocol. @sameo Do you have any opinion on this?
@Xynnn007 Thanks a lot for putting such a detailed and thorough issue.
As @jialez0 I prefer the first option, and as a matter of fact the path-based scheme is already implemented.
I dont think we're going to need anything else from the KBS implementation (except from maybe defining an architecture and API to add new resources) and most of the remaining work is on the image-rs and AA/KBC side of things. This proposal (based on the first option) will decouple the KBC image creation process from its unwrapping, so it's definitely needed. It's a big +1 from me.
Great proposal. I just have a few comments.
First, once we adopt a universal resource id, it might seem like the unwrap_key
and get_resource
endpoints will do the same thing. From the KBS perspective this is true, but the API of the AA is different because ocicrypt-rs
is expecting to use the keyprovider API. It's fine to have two different APIs for keys and other resources, but there are a couple things to think about. First, does this protocol support people who might want to use truly remote key unwrapping. As in, some people might not want the key that unwraps the annotation data to leave their KBS. Instead they might want to send the entire annotation to the KBS and get the unwrapped version in response. Would we be able to do that? On the other hand, it might be worth thinking about ditching the keyprovider API altogether. We've talked about unifying the container encryption scheme. One way to do this would be to move the unwrapping into ocicrypt-rs
, which would only use the get_resource
endpoint to request individual keys. This would be a pretty big change (and it wouldn't be compatible with the remote unwrapping thing I mentioned before), but it's something to think about.
Second, I am a little bit wary of having the KBS type in the URI. In general we want containers to have as broad compatibility as possible. I see that you are proposing having a generic KBS type as well, which could be useful. In some ways it might be better to specify which KBS is expected than to have silent failures. I'm not exactly sure what the right approach is yet. I think that we will be able to support this URI in the simple-kbs
with very minimal changes although currently we have no concept of a repository, only a type and a tag. We could add this or we could just ignore the repository. The question of whether one KBS is compatible with another is kind of complicated. Like I said, I'm not sure the best solution, but I am wary of hardcoding the KBS into the image.
Another question is how the KBS should handle access control for resources. The KBS should probably have some mechanism for allowing certain guests to access certain resources (based on the result of the AS). Should this be individual resource level or more general (i.e. the repository level). For simple-kbs
we allow every resource to have its own policy for when it should be released, but this is probably overkill and with multiple platforms the configuration space grows significantly. On the other hand we don't want every resource to be available to anyone with a valid attestation.
Hi @fitzthum , thanks for your deep thinking about the proposal! I'm very happy to share my views.
Surely unwrap_key
and get_resource
are doing some overlapping things, but I think it is reasonable to keep both because unwrap_key
follows ocicrypt
and might not expose the key (resource) out of KBS, while get_resource
should expose the resource out of KBS and only defined in CoCo.
First, does this protocol support people who might want to use truly remote key unwrapping. As in, some people might not want the key that unwraps the annotation data to leave their KBS. Instead they might want to send the entire annotation to the KBS and get the unwrapped version in response. Would we be able to do that?
Yes, you're right and we can as we do currently. When we changed API from decrypt_payload
to get_key
, we found that KBSes using some kinds of KMS as backend did not support export the key, making decryption only happen inside KBS.
On the other hand, it might be worth thinking about ditching the keyprovider API altogether. We've talked about unifying the container encryption scheme. One way to do this would be to move the unwrapping into
ocicrypt-rs
, which would only use theget_resource
endpoint to request individual keys. This would be a pretty big change (and it wouldn't be compatible with the remote unwrapping thing I mentioned before), but it's something to think about.
I may have some objections. The reasons are
KeyProvider
is defined by ocicrypt
. If we keep using it there will be compatibility.ocicrypt-rs
only calls gRPC of KeyProvider
, and send the AnnotationPacket
to AA for decrypting. As you've mentioned, some KBS does not expose the key outside, we might not be able to request individual keys.KeyProvider
and GetResource
api. KeyProvider
follows ocicrypt
and send the whole AnnotationPacket
(which should include the identity of decryption key) to AA's KBC to decrypt PLBCO. GetResource
only contains the identity of the resource requested, which needs to do some change later as current parameters can be replaced by a single url of resource
. Both apis are based on the remote attestation happened before.Besides, as that image encryption in ocicrypt-rs
, cosign image signing in sigstore-rs
are prepared, we think it is time to provide a integration tool when generic KBS is prepared. The tool will encrypt (in ocicrypt way) & sign (in cosign way) the image and registry the KEK into KBS.
Second, I am a little bit wary of having the KBS type in the URI. In general we want containers to have as broad compatibility as possible. I see that you are proposing having a generic KBS type as well, which could be useful. In some ways it might be better to specify which KBS is expected than to have silent failures. I'm not exactly sure what the right approach is yet. I think that we will be able to support this URI in the
simple-kbs
with very minimal changes although currently we have no concept of a repository, only a type and a tag. We could add this or we could just ignore the repository. The question of whether one KBS is compatible with another is kind of complicated. Like I said, I'm not sure the best solution, but I am wary of hardcoding the KBS into the image.
I agree with you. Hardcoding the KBS into the image includes two aspects:
kbs://example.cckbs.org/my/Cosign-key/1
, it should only care about my
, Cosign-key
and 1
but ignore example.cckbs.org
as this KBC does not connect to a remote KBS. In this way, we should do some refactoring to all kinds of KBCs.quay.io/alpine:latest
and docker.io/alpine:latest
. Also it implies that an AA can serve to connect different KBS, thus we do not need to input the kbs uri from kata side in future. Certainly the identity of KBS (pubkey cert distribution) is a problem to resolve.Another question is how the KBS should handle access control for resources. The KBS should probably have some mechanism for allowing certain guests to access certain resources (based on the result of the AS). Should this be individual resource level or more general (i.e. the repository level). For
simple-kbs
we allow every resource to have its own policy for when it should be released, but this is probably overkill and with multiple platforms the configuration space grows significantly. On the other hand we don't want every resource to be available to anyone with a valid attestation.
Exactly! Overall, I agree with you. I think (pod/user) identification and access control are another problem to handle, which is related to the identity of TEE/tenant/container/pod we can talk in another proposal/issue. There are some initial thinking I want to share:
pod/deploy
, TEE
, container
and user/tenant
.Second, I am a little bit wary of having the KBS type in the URI. In general we want containers to have as broad compatibility as possible. I see that you are proposing having a generic KBS type as well, which could be useful. In some ways it might be better to specify which KBS is expected than to have silent failures. I'm not exactly sure what the right approach is yet. I think that we will be able to support this URI in the
simple-kbs
with very minimal changes although currently we have no concept of a repository, only a type and a tag. We could add this or we could just ignore the repository. The question of whether one KBS is compatible with another is kind of complicated. Like I said, I'm not sure the best solution, but I am wary of hardcoding the KBS into the image.
Now a kata-agent will be given the type of KBC, address of KBS. Then it will call inside image-rs for resources, however, only to one KBS specified by launch parameter of kata-agent. I think this can be better. As actually the kata-agent does not need to know which KBS or KBC the user will use. As for the problem of distribution of KBS public key, the pubkeys are distributed from the user. The user can give many public keys, which share the same logic of one key, even if some of them will not be used.
In this way, type of KBC and the address of KBS should be included in a URI where the resource is used, by which we can reduce another parameter in the startup parameters of kata-agent, and let the image-rs to flexibly support request resources from different KBS. There is another problem if we keep a universal kbs://
, that the AA logic does not know which KBC should be dispatched to.
I do not get a good way to deal with this, up to now -
Hi @Xynnn007. Thank you for the detailed write up and rationale.
Overall, I like it, and I too have a preference for option 1. However, I am not convinced by the rationale for needing different kbc types. Isn't the resource type sufficient for the KBS to figure out if it's something it knows how to deal with? Normally the "schema" portion of a URL is intended to describe the type of protocol, and as long as you expect the underlying protocol to be the same, I think that you should use a single schema for simplicity. You suggested yourself using 'kbs' to cover all the KBCs, and I believe we should aim for that from the start, with a well-defined response when the KBS does not know how to deal with a given resource type.
Hi @c3d thank you very much for your comment. Please check the previous comment of your comment. To sum up, currently the KBS info is given from kata-agent, which I prefer to abondon to include this information in the uri of the resource. In this way we can support to request resources from different KBSs.
If we do not need this for AA, a universal kbs://
is good.
@Xynnn007
As I've mentioned before, I think getting resources from multiple KBSes should be thought of more as an anti-pattern than something we should support. There are some very tricky security issues that come up when we do this.
Even if we just have one KBS, it's hard to make sure that we are talking to the right one. Putting the public key of the KBS in the measurement does not solve this problem, because a malicious KBS could simply ignore this part of the measurement. I write about this at length here. Getting signatures and secrets from different KBSes is particularly dangerous as mentioned in https://github.com/confidential-containers/guest-components/issues/194.
You mention that we could give multiple KBS public keys to support multiple KBSes, but again this is not as reliable as it seems and even if it was, then each KBS would have to know the public keys of all the other KBSes that it trusted in order to validate the measurement.
Is it possible to have multiple KBS URIs specified in a configuration, and specify a priority for which URI should be used primarily?
@fitzthum wrote:
As I've mentioned before, I think getting resources from multiple KBSes should be thought of more as an anti-pattern than something we should support. There are some very tricky security issues that come up when we do this.
As discussed in today's meeting, there may be legitimate contexts where we actually need multiple KBS's, notably in teh case where a device like a vGPU needs its own secrets from the KBS. The consensus in today's meeting seems to have been that any encrypted communication between CPU and GPU could be managed with a Diffie-Hellman key exchange. However, I could think of things like license keys (NVIDIA has been known to require specific licenses to enable some features) that could come from a KBS.
So while I agree that having multiple KBSs being able to deliver the same secrets looks like something we should prevent as much as possible, having multiple KBSs that deliver secrets to different components might be a valid option. So I'm wondering if it's possible to have something in the URI that let's us distinguish between valid and invalid configs. Like having some sort of format convention for all URIs where we could say "if this or that part of the URI is the same, then we can safely reject".
@hdxia please see if this would be compatible with your proposed KBS.
@c3d
So while I agree that having multiple KBSs being able to deliver the same secrets looks like something we should prevent as much as possible, having multiple KBSs that deliver secrets to different components might be a valid option. So I'm wondering if it's possible to have something in the URI that let's us distinguish between valid and invalid configs. Like having some sort of format convention for all URIs where we could say "if this or that part of the URI is the same, then we can safely reject".
It's not as simple as avoiding delivery of the same secrets. The most important thing is probably to make sure that signature validation information comes from the same source as any sensitive/secret material. In general we want to enforce that an enclave belongs to only one client. We can run into major problems if we start mixing secrets and policies from different parties.
I think it might be hard to enforce this at the level of the resource URI. We don't really control the names of resources. They could even use two different names to identify the same resource. I'm not sure how we would detect this.
Fortunately I think the GPU stuff is operating at a slightly different scope. If there is some component inside the guest that needs to connect to a license server as part of GPU attestation, I think that is fine. It doesn't need to interact with the KBS at all and shouldn't compromise any of our guarantees. My thinking is that as long as we are measuring whatever this component is, we can more or less let it be.
Background
Currently, CC Key Broker System (including CC-KBC and Generic KBS) is under development. At the meantime, we have started to use other Key Broker Systems (like Offline-FS-KBC, EAA-KBC & Verdictd to distribute some secret resources (key resources, like policy.json for image security, image signature verification key, image registry credential, etc.). However, we still need an identifier to uniquely identify a specific KBS resource, because without that mechinism, for example, we cannot request different image verification keys inside one Pod from KBS. Let's explain it.
Currently, the way we use different KBS, is to hard-code a key which is used to identifier the resource inside the code, like policy.json and cosign verification key in image-rs. Then, we prepare different resources for each hard-coded key in experimental KBSes, such as resources.json for offline-fs-kbc and cosign verification key, policy for verdictd. Here the description granularity of the key here is very rough, which means that if we want to request a public key, there will be no identifier to determine which public key is required, thus making it unable to support multiple public key in both image-rs and KBS side.
Similar problems also occur in Credentials of image Registry, image security policy files, etc.
Detailed Description of Current Implementation
Let's take the scenario where using image-rs to request the public key from KBS (Verdictd) to perform image signature verification as an example.
What we do now is to hard-code a constant
"Cosign Key"
inside image-rs. Every time image-rs needs to request KBS for a public key, it will do the following steps:"Cosign Key"
to construct aResourceDescription
struct{"name": "Cosign Key"}
ResourceDescription
as input parameter to request GetResource gRPC API of Attestation-Agent (AA for short in the following)ResourceDescription
. Then it follows its own logic to access Verdictd on the other node to get the public key.The problem here is as mentioned:
Similiar resources includes credentials to access private registries, etc. If there are more and more resource types, and every resource has different instances on the KBS side, it seems impossible for us to use hard-code to meet this need.
To resolve this problem, we propose a KBS Uniform Resource Locator (URL) protocol. We can use KBS URL together with configuration file to support image-rs and other components to retrieve specific resource on the specific KBS server.
Let's quick go through how KBS URL can work with CC-KBC and generic-KBS.
About CC-KBC and generic-KBS
We hope that new KBS URL can work well with CC-KBC and generic-KBS. Let's take an example, still cosign public key. The diagram below shows each entity that carries the identity information of each call routine without KBS URL.
Cosign Key
field.If we bring KBS URL and configuration file in, the workflow will be
Target
This proposal aims to
This is especially critical for many scenarios, like
key_id
field of the annotation packet of the layer.KBS URL Protocol
There are two options to define KBS URL Protocol.
Option 1
This option uses path-like scheme to define KBS URL. The format is as following
It seems like a url for image, which has scheme
<registry>/<repository>:<tag>
. Here,<kbc-type>
: KBC Type to be used to access the target endpoint, likecckbc
,eaakbc
, etc.<url-of-kbs>
: KBS server's url. It implies that KBS OpenAPI and Protocol is support by the server.<repository>
: A parent path. Usually, we can use an username or something meaningful.<type>
: To distinguish different resource types.<tag>
: To distinguish different resource instances.For example, Alice has an instance
key1
of resource typeCosign-Key
on KBS serverexample.cckbs.org
. We can usecckbc://example.cckbs.org/alice/Cosign-Key/key1
to uniquely identify the key.Option 2
This option uses query-like scheme to define KBS URL. The format is as following
Here,
<kbc-type>
: KBC Type to be used to access the target endpoint, likecckbc
,eaakbc
, etc.<url-of-kbs>
: KBS server's url. It implies that KBS OpenAPI and Protocol is support by the server.<keyi>=<valuei>
: A group of query conditions concatenated by&
Here, the query conditions can be freely set by the user. For example:
cckbc://example.cckbs.org/user=Alice&resource=Credential
cckbc://example.cckbs.org/user=Bob&resource=public-key&keyid=321
cckbc://example.cckbs.org/type=policy&id=abc
I Prefer to Option 1, because it seems more tight, and somehow we may not need very flexible query machinism like option 2. Please share your opinions.
About KBC Types
Although there are some different KBC types, like cc-kbc, offline-fs-kbc, etc.
So, we can define
<kbc-type>
field like the following:eaakbc://
eaakbc & verdictdofflinefskbc://
offlinefskbccckbc://
cckbc & generic-KBSsamplekbc://
sample kbcofflinesevkbc://
offline sev kbconlinesevkbc://
online sev kbcHere we distinguish different
kbc
's protocol because different kbc may handlekey_id
in different way. And we may go on a talk whether we agree on a uniform protocolkbs://
to cover all the mentioned kbcs.Use cases
This section clearifies the producer and the consumer of KBS URL.
Producer
Here are two examples.
Configuration of Image Security
policy.json
We can use KBS URL in the policy.json to specify a public key to verify the given image like the following (using Option 1 for example)
We can use key
"cckbc://example.cckbs.org/my/Cosign-key/1"
for imagequay.io/kata-containers/confidential-containers:cosign-signed-1
and key"cckbc://example.cckbs.org/my/Cosign-key/2"
for imagequay.io/kata-containers/confidential-containers:cosign-signed-2
when doing signature verification.Annotation Packet Generated by CC-KBS When Encrypting an Image
In current CCv1 image security design, there is a entry with key
"org.opencontainers.image.enc.keys.provider.attestation-agent"
in the manifest of a encrypted image layer. The entry's value is calledAnnotation Packet
. TheAnnotation Packet
contains the encrypted LEK (Layer Encryption Key, the key to encrypt the layer) by a KEK (Key Encryption Key, the key to encrypt LEK), an id of the KEK of some KBS, like the followingHere
"key_id"
specifies the concrete KEK inside a given KBS. However, this way might only fit in experimental environment, because the owner of the image might have multiple KBS. We bring in KBS URL here similarly, s.t.Now when decrypting the layer, it will be more conrete to know where the decryption key is.
Consumer
Image-rs
The KBS URL in the first example above is directly consumed by image-rs, following the steps:
ResourceDescription
in a determined wayResourceDescription
to request AA'sGetResource
gRPC APICC-KBC
The KBS URL in the second example above is consumed by CC-KBC, following the steps:
Annotation Packet
from the manifest of the image.UnWrapKey
gRPC to send theAnnotation Packet
to AA.Annotation Packet
and getkey_id
field.<kbc-type>
, AA will deliver thekey_id
to related KBC plug-in. In this case it will becc-kbc
, and CC-Kbc will convert the KBS URL into a OpenAPI Web URL of KBS, s.t. fromcckbc://example.cckbs.org/my/key/1234
to something likehttps://example.cckbs.org/kbs/v0/key?repository=my&type=key&tag=1234
(This web url depends on how KBS OpenAPI defines)Behavior in Kata-CC
Fow now, only one KBS is used by a pod in Kata-cc and its address is delivered by a config file. So we can do some specific thing for kata-cc:
<kbs-url>
subfield of the KBS URL with the one in the config, and then deliver the modified one to AA.Influence
To KBS Protocol
Way 1: Without modifying OpenAPI definition
We can implement two functions:
cckbc://
to a web url of KBS OpenAPI Http Server, which can be used in cc-kbcStill, some suggestions for KBS protocol should be considered in Way 2
Way 2: Some modification for OpenAPI definition
KBS protocol is defined by OpenAPI. Currently, apis for resource retrievement are two:
/resources/key/<key_id>
: key resources/resources/token
: tokenWe suggest to expend the definition, s.t. use
resource
to descript all kinds of resources includingkey
,token
, etc. How to process is implemented by the KBS server itself. In this way, we can have two strategies to modify the API design due to differernt choice of KBS URL option.Path-like KBS URL
For KBS URL using option 1, s.t.
<kbc-type>://<url-of-kbs>/<repository>/<type>/<tag>
The definition of
resource
OpenAPI can be something likeAnd the url of HTTP GET request will be like
https://example.cckbs.org/kbs/v0/resources/alice/Credential/database
Query-like KBS URL
For KBS URL using option 2, s.t.
<kbc-type>://<url-of-kbs>/<key1>=<value1>&<key2>=<value2>&...
The definition of
resource
OpenAPI can be something liketo support free query form. For example, the requested url can be
https://example.cckbs.org/kbs/v0/resources?repository=alice&type=Credential&tag=database
https://example.cckbs.org/kbs/v0/resources?name=bob&id=233
To image-rs
Signature Verification
Now, a policy.json for image-rs looks like
It means that the public key is in path
"/run/image-security/cosign/cosign.pub"
. If there is not, image-rs will request KBS for"Cosign Key"
. If we bring KBS URL in, arbitrary resource can be used. The policy can beMeans that this image needs a public key tagged
1
in the repository namedmy
of generic-KBS serverexample.cckbs.org
.The same mechinism applies to registry credentials, policy.json, etc. In future, more scenarios could use this url mechinism for not only image-rs.
References
https://github.com/apigee/registry/blob/main/openapi.yaml https://spec.openapis.org/oas/latest.html#paths-object https://github.com/confidential-containers/image-rs/issues/50 https://github.com/confidential-containers/guest-components/issues/218