Proposal: Non-intrusive Custom KeyWrap Protocols

[lumjjb]

OCICRYPT dynamic custom keywrap protocol support

This proposal is to add the ability to support custom keywrap protocols with minimal to no code changes to downstream consumers of the ocicrypt library and to allow custom protocols to be implemented without changes to ocicrypt.

Keywrap protocols such as "org.opencontainers.image.enc.keys.custom.*". Examples are:

"org.opencontainers.image.enc.keys.custom.isecl"
"org.opencontainers.image.enc.keys.custom.keyprotect"
"org.opencontainers.image.enc.keys.custom.azurekeyvault"

The end result should provide the ability to configure the library and all downstream users of it via an environment variable that points to a configuration file. This config file will then provide the information needed to perform call out to executables to perform the keywrapping/unwrapping

Example Usage/Configuration

The following are examples of downstream usage:

Example of config

The config file /etc/ocicrypt would look something like:

"custom-protocols": {
    "isecl": {
       "cmd": "/usr/lib/ocicrypt-isecl",   
       "args": []
    },
    "keyprotect": {
       "cmd": "/usr/lib/ocicrypt-keyprotect",   
       "args": []
    },
    "keyvault": {
       "grpc": "unix://run/myapp.sock"
    }
}

Env variable config reference

The config file would then be referenced via environment variable OCICRYPT_CUSTOM_CONFIG:

OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json skopeo copy ...
OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json buildah push ...
OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json buildah pull ...
OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json ctd-decoder ...
OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json crio ...

Passing of encryption/decryption keys

Passing of encryption and decryption keys would be implemented via "custom:" prefix, followed by the named prefix of the protocol, for example, the protocol "org.opencontainers.image.enc.keys.custom.isecl" would appear like the following:

OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json skopeo copy --encryption-key custom:isecl:some-params

The same would follow for decryption config

OCICRPYT_CUSTOM_CONFIG=/etc/ocicrypt_custom.json skopeo copy --decryption-key custom:some-params

Underworkings and interfaces

Implementation/Integration points

There are two main implementation points. These are passing of encryption/decryption parameters, as well as the handling of the keywrap interface.

Passing of parameters

Most downstream implementations make use of the CreateCryptoConfig helper to parse encryption/decryption parameters. These would have to be moeified to translate the parameters provided for encryption/decryption to part of a CryptoConfig. No parsing or reading of files should be done at this stage.

This should be able to be handled by the custom keywrap implementation. The parameters will be stoed in the parameter maps with the keys of their prefix, i.e. parameters["custom:secl"] = "some-params".

Handling of keywrap interface

Implementing a "custom" KeyWrap implementation

A KeyWrap interface needs to be implemented called "custom". This implementation would need to be able to look up the environment variable for the config and parse it.

It should then take the inputs of the keywrap interface and call the associated variable, passing in the required arguemnts of the keywrap/unwrap interface as serialized JSON in STDIN.

The custom keywrap implementation will call into the executable passing in json.Marshal(CustomKeyWrapProtocolInput{...}), and parse the exit code of the call for error, if exit with non-zero, treat it as an error and take STDERR as the output err message. Else, parse STDIN as json.Unmarshal(..., &CustomKeyWrapProtocolOuput{}) and pass data base as according to the operation performed.

We will define the following structs as an interface:

type CustomKeyWrapProtocolOperation string

var (
   OpKeyWrap CustomKeyWrapProtocolOperation = "keywrap"
   OpKeyUnwrap CustomKeyWrapProtocolOperation = "keyunwrap"
)

type CustomKeyWrapProtocolInput struct {
    // Operation is either "keywrap" or "keyunwrap"
    Operation CustomKeyWrapProtocolOperation `json:"op"` 
    // KeyWrapParams encodes the arguments to key wrap if operation is set to wrap
    KeyWrapParams KeyWrapParams `json:"keywrapparams",omitempty`
    // KeyUnwrapParams encodes the arguments to key unwrap if operation is set to unwrap
    KeyUnwrapParams KeyUnwrapParams `json:"keyunwrapparams",omitempty`
}

type CustomKeyWrapProtocolOuput struct {
    // KeyWrapResult encodes the results to key wrap if operation is to wrap
    KeyWrapResults  KeyWrapResults `json:"keywrapresults",omitempty`
    // KeyUnwrapResult encodes the result to key unwrap if operation is to unwrap
    KeyUnwrapResults KeyUnwrapResults `json:"keyunwrapresults",omitempty`
}

type KeyWrapParams struct {
    Ec *config.EncryptConfig `json:"ec"`
    OptsData []byte `json:"optsdata"`
}
type KeyUnwrapParams struct {
    Dc *config.DecryptConfig `json:"dc"`
    Annotation []byte `json:"annotation"`
}

type KeyUnwrapResults struct {
    OptsData []byte `json:"optsdata"`
}
type KeyWrapResults struct {
    Annotation[]byte `json:"annotation"`
}

Cmd callouts vs gRPC

Generally cmd call outs are simpler to use, however, in managed kubernetes clusters, there is a high barrier to installing software on the host where the container runtime resides. The gRPC approach would allow a daemonset to be run to server gRPC calls with the same structs (converted to gRPC as regular JSON types would be).

Adding scheme lookup special case for custom protocols

The GetKeyWrapper function for resolving the keywrapper to use needs to handle the special case of custom protocols as we are routing to the custom keywrap implementation based on just the prefix "org.opencontainers.image.enc.keys.custom", instead of the fully qualified annotation string.

https://github.com/containers/ocicrypt/blob/master/encryption.go#L61

Likewise, any code that picks up annotation or wants to write to them should handle this new special case as well

https://github.com/containers/ocicrypt/blob/c835e1c1df9806083f1d2f51c17d2fc944bf8c8e/encryption.go#L135

NOTE on implementation: This could be done as a golang init() function that reads the config file as well, implementation is subjective to whichever is cleaner. This means that any long-running processes will require reload if config file changes though.

[stefanberger]

Following the recent extensions with pkcs11, we now have the following in a config file referenced by the OCICRYPT_CONFIG environment variable:

pkcs11:
  module-directories:
  - /usr/lib64/pkcs11/
  - /usr/lib/softhsm/
  allowed-module-paths:
  - /usr/lib64/pkcs11/
  - /usr/lib/softhsm/

If we now extend this with what you have above it could look like this:

pkcs11:
  module-directories:
  - /usr/lib64/pkcs11/
  - /usr/lib/softhsm/
  allowed-module-paths:
  - /usr/lib64/pkcs11/
  - /usr/lib/softhsm/
custom-protocols:
  - isecl:
    cmd: /usr/lib/ocicrypt-isecl
    args:
  - keyprotect
    cmd: /usr/lib/ocicrypt-keyprotect
    args:

I think we should try to have this one one configuration file. The special configuration of OCICRYPT_CONFIG=internal would then not be usable if custom protocols were also in use.

[lumjjb]

The special configuration of OCICRYPT_CONFIG=internal would then not be usable if custom protocols were also in use.

We could add a field within pkcs11 to set internal_config = true/false to perform that behavior?

[lumjjb]

I do think that there is also merit in separating out the two configs, so that technically if one has limitations that it cant be dynamic throughout the lifetime of the program (like pre-registering the custom protocols in the keywrapper in the init() function), they could be treated with separate semantics.

[hdxia]

Instead of using "custom", for example, "org.opencontainers.image.enc.keys.custom.isecl". I feel it is better to use "provider", like "org.opencontainers.image.enc.keys.provider.isecl". this is similar to Kubernetes secret KMS provider. if the way proposed is to be included as part of ocicrypt, it is not "custom" anymore. the keyword used in example need to be changed to "provider" too. Another consideration is should we use unix socket, like what Kubernetes KMS provider does as well, just leverage it?

[stefanberger]

Another consideration is should we use unix socket, like what Kubernetes KMS provider does as well, just leverage it?

You mean unix sockets to pass parameters? I think the plan was to use pipes and passing the read-end of a pipe to the launched command line tool, e.g. /usr/lib/ocicrypt-keyprotect, to have it read a SON object.

[hdxia]

yeah, just for consideration. the binary will be simple with pipes I think. the kms provider for kubenetes secret example below

apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources:

• resources:
  • secrets providers:
  • kms: name : myKmsPlugin endpoint: unix:///tmp/socketfile.sock cachesize: 100
  • aescbc: keys:
    • name: key1 secret: <BASE 64 ENCODED SECRET>

[lumjjb]

@hdxia thanks for the comments and feedback!

Instead of using "custom", for example, "org.opencontainers.image.enc.keys.custom.isecl". I feel it is better to use "provider", like "org.opencontainers.image.enc.keys.provider.isecl".

I like this suggestion, provider sounds like a good name.

Another consideration is should we use unix socket

I am not that big a fan of the unix socket here. I think that sockets would be appropriate if there is a good reason to have a daemon based provider. Having a daemon introduces some complexity and unless we have a good reason to, we should avoid it. I think stateless calls to the binary with serialized input/output should work well enough for us.

Configuration

I am thinking with regards to the configuration required, that if there are additional parameters that the binary needs to have, that they can be passed with the args field. Will this work for what you would like to do?

custom-protocols:
  - isecl:
    cmd: /usr/lib/ocicrypt-isecl
    args: ["--cert", "cert.pem", "--wlagent/wpm-path", "..."] 

[rajeshs-entrust]

Running the keywrap as an executable is probably okay. However, running the decryption module/provider on the host might be a problem as it has to run on the worker nodes. RHCOS discourages installing software on the worker nodes. Running the decryption provider as a container would make it easy to deploy the decryption provider (e.g. as daemon set). (G)RPC interface would be preferred. Any thoughts?

[hdxia]

Running the keywrap as an executable is probably okay. However, running the decryption module/provider on the host might be a problem as it has to run on the worker nodes. RHCOS discourages installing software on the worker nodes. Running the decryption provider as a container would make it easy to deploy the decryption provider (e.g. as daemon set). (G)RPC interface would be preferred. Any thoughts?

This is a good point with the restriction of RHCOS. then it goes back to the unix socket approach-grpc.

[lumjjb]

Ok I think this is a good enough reason to, I think we can support both execve call outs and gRPC. Exec call outs are going to be useful for the build side and gRPC will be useful for managed deployment cases where there is less control of the host.

EDIT: @bkatchapalayam-hytrust @hdxia Amended the doc to include gRPC use as well

[lumjjb]

Resolved by https://github.com/containers/ocicrypt/pull/38. Thanks @pravinrajr9 !