Generalized Notary Client External Signer/HSM Interface

[dbilling]

I'm new to TUF and notary but I'm familiar with the corporate software signing operations at the company where I work. We use a network-based HSM solution for signing that handles our large organization problems - problems like corporate access control and authorization of signing requests, ability to do signing requests from data center VMs that don't have accessible usb ports, etc. etc.

As we investigated signing for docker images we've found the most important private keys are stored on the notary client machine or a local yubikey. Neither of these solutions satisfies the signing needs of a large organization like ours. I looked through other notary issues and I am excited to see several folks talking about extending the keystore/HSM support on the client. However, unless I'm mistaken (which is entirely possible), it appears the writers of these issues propose to modify notary client to work with their specific HSM and/or their specific PKCS#11 interface package. This is all good, but takes notary down a (never ending?) path of supporting all sort of different variants of HSMs and HSM interfaces inside the client. Might it be better to have one external interface for all this?

So, what I'm proposing here is a generalized "external keystore" plugin solution -- some sort of proper interface/API where any interested party could write a plugin to teach the notary client to interact with their specific HSM/keystore needs -- such that the code that interfaces with that keystore/HSM would be outside notary. The interface wouldn't be PKCS11 or KMIP or MS-CAPI -- instead it would be a notary defined plugin interface anyone could write a plugin to for whatever local or remote HSM/keystore they need accessible from the notary client. With such an API in place, the notary client would instantly have a compatibility story for interacting with any local or remote keystore/HSM/HSM system without having to muck with notary itself.

Thoughts? After writing all this I found the trustmanager remoteks grpc interface, but that seems to be solving a different problem. Could it be used/extended/adapted for this purpose?

[justincormack]

Hmm, I think the hope was that PKCS11 was standard enough to be supported by the devices people use, and that the current work on restructuring that would make it easier to use with other devices. What APIs are you interested in supporting? I have had recent interest in better HSM integration just yesterday, so interested in this area.

[dbilling]

PKCS#11 is fine for communicating with local HSM hardware. However, with the advent of this cool thing called the network, commercial enterprise-level HSMs systems aren't typically local to the machine that needs the signing done. Those sorts of systems will typically want KMIP. Then there's folks like me that have to interact with highly customized systems. I will be coding to a RESTful API for HSM access. The point is, there's a whole world here. I'm trying to think strategically and get the complications of every HSM protocol and environment out of the notary client -- create a higher level API and let the community take care of it. That's why I used words like "plugin" in the write-up above. What's the easiest way to have an interface out of the notary client and into a 3rd party provided local app that handles all the specifics of their HSM/keystore setup? That's what's needed.

[endophage]

@dbilling if you're planning for a RESTful API, you might instead want to look at the GRPC based remote signer interface we've already defined: https://github.com/theupdateframework/notary/tree/master/trustmanager/remoteks

I apologize that I never really got around to writing up docs for it. I'll look into that over the upcoming labor day weekend.

[endophage]

Actually, I'm looking at that again and it's more about key escrow. It implements the trustmanager.Storage interface. I'd intended but I guess never got round to implementing a similar GRPC interface for the trustmanager.KeyStore interface. In that instance the concrete underlying struct that implemented the PrivateKey interface would make additional GRPC calls when its Sign method was called.

I feel like the code for that may be sitting in a stale PR and never got merged due to lacking priority. I'll see if I can dig it up

[dbilling]

Hi @endophage, I'm a GRPC newbie, but this all sounds super interesting and I'd be quite happy to learn and use GRPC for my interoperability case if that's how you experts want to handle the "how do we let others plugin their network-based HSM" problem for the notary client. It does look from your second post like the current support for GPRC solves a different issue and is not quite enough to handle the keystore case, if I read all that correctly. Let me know what you need from me... I'd be happy to reorganize/repurpose this issue for that support, close this one and open another, and/or help in anyway necessary make this happen...

[endophage]

GRPC is ultimately implemented over HTTP2. I see it as providing 2 key advantages over designing and building a RESTful API directly:

1. Clear definition of the available "routes" in the .proto file.
2. Compilation of client and server libraries in many languages, making integrations faster and more consistent.

The current state is not that GRPC itself solves a different problem it's that we haven't yet defined the GRPC interface and integrated it into notary for your specific use case (and we did intend to support your use case at some point).

This issue hasn't gotten too long so I'm happy to keep this as the issue for defining the remote signing interface. Any existing code would likely be in this PR and there would be a distinct proto file defining the remote signer interface. As I'm not seeing that proto file I'm pretty sure there's no existing code.

[dbilling]

Sounds like we have a good plan on how to solve the problem, but there's no existing work/stale PR for this specific grpc use case. @endophage, is this something you'd prefer to code up or would you rather I put together a contribution?

[endophage]

@dbilling to keep it focussed, why don't you start by proposing a .proto definition for the GRPC interface that would back a remote instance of the KeyStore. The GRPC interface would necessarily also implement the PrivateKey interface.

I'd anticipate GetKey would essentially be a quick lookup on the KeyID and would return a pseudo-PrivateKey that holds the KeyID and passes through calls to Sign to the GRPC service.

[dbilling]

@endophage @justincormack, I've taken a look at this and I have a proposal grpc .proto to share...what's your preferred method for starting a discussion/getting feedback on the .proto file? Along the way, I've come up with the following questions:

1) In truly secure environments, HSMs generate key pairs and never expose the private key at all. Can the current trustmanager.KeyStore interface handles this case? If not, do you have a suggested solution to the problem where public key/pseudo private key needs to get into the notary client from the HSM but the private key material never leaves the HSM, ever. 2) Does notary require a one-to-one mapping between a keyinfo (GUN/role) and a key pair? or can multiple GUNs share the same key? GetKeyInfo() seems to imply that each key pair is tied uniquely to one GUN/role, but the rest of the interface seems to specifically allow for multiple GUNs to share a key. 3) What key types/signature types need to be supported over the grpc interface? obviously the most important is ecdsa/ecdsa, but if other key types/sig types are necessary across this interface pls let me know the list.
4) Similar to above, within ecdsa, should the interface support different curves? 5) A Somewhat related question: I can't figure out how someone would move trust data between notary servers without resigning. The use case here for docker is: a docker image is posted to docker.io (registry A) and signed by "Author". Some consumer of this image ("Joe") wants a copy of the image in a different registry (registry B). In the docker world, he'd do a pull from A, tag for B, and a push to B. However, Joe wants to keep the original trust data intact (he wants the unmodified image in registry B to still be signed by "Author", not by "Joe". Is this possible? Such a feature that allows for trust replication will be critical when trust pinning is finally arrives in docker.

[endophage]

@dbilling We generally use PR's for any type of code review, even early stage proposals. Regarding your other questions:

1. KeyStore can't handle this today. I'd suggest creating a new interface something like KeyGenerator and having the concrete HSM KeyStore implementation define that method. The Create method on CryptoService could then iterate the KeyStores attached to the CryptoService and do a type check to see if any of them implement KeyGenerator, falling back to its current behaviour if none of them do.
2. The tight coupling of keys to guns is for the most part unintentional tech debt. It seemed like a better idea 3 years ago than it turned out to be. There are some security benefits in not re-using keys, but it also means most users have to juggle a lot of keys, which comes with its own set of security downsides. I'd want the input of the other maintainers, but I'd be more than OK with starting to unwind the coupling.
3. I'd want to see at least one RSA 4096 based scheme supported in addition to ECDSA.
4. Possibly making point 3. moot, is it possible to make the interface indifferent to the algorithm?
5. The CLI doesn't really make it possible today (in this case there's definitely some code in a PR somewhere), but a bit of curl and bash can absolutely do it. If you pull the current trust data from one server and upload those files as a multi-file upload to the other under the same name, it'll work fine. The issue we've historically run into with Docker itself is that image names and registry locations are tightly coupled. I think there has been some progress on that front in Docker but I don't have anything specific to point to (maybe @justincormack can help there?).

[justincormack]

1. Yes we definitely want to be able to support interfaces that can't expose private keys.
2. Agree with David, starting to change this would be good.
3. RSA 4096, ed25519, ECDSA p256 cover most cases (although there are still two padding schemes for RSA that are common).
4. Support for different curves in ECDSA is not commonly used. While it would be nice to make the interface indifferent, I think there are HSMs that require the format to be specified so we probably need a list.
5. We are trying to fix the naming issue with Docker, and are planning more work towards this, as it is a problem. There is some mirroring configuration support at present but it needs extending to support authentication better, and further extensions to decouple naming, but we want to resolve this soon as it is very limiting.

[endophage]

I should have been more clear re. 1. We already support not exposing private keys for signing. It's only the creation that needs to be updated. The existing Yubikey code returns a type that implements the PrivateKey interface but can't actually return the private key bytes. It simply allows you to call Sign which passes through to the Yubikey's signing functionality.

Also, @justincormack, re 4. want to make sure we're talking about the same thing. I'd want to see if we can make the notary interface generic. A concrete implementation of an HSM KeyStore could of course pass a format. Could we hide that format behind the PrivateKey interface so it gets injected as part of the Sign operation but the user never has to think about it? That might require Notary enforces a single format per key type, but that doesn't seems like a bad thing.

[alukyan]

@dbilling - thanks for this PR and your work - controlling and auditing access to key material is super-important in cloud automation scenarios. Looking forward to see this PR get merged. I am interested in this since this can be used to use cloud key vault (Azure Keyvault) to store notary signing keys to satisfy security and compliance requirements when using Docker Content Trust. Would be great to have reference implementation of remote keystore server.

[dbilling]

Hi @alukan - Thank you for the encouragement. I am motivated to continue to move this toward a merge. In regards to your reference implementation question, The trustmanager/grpckeystore/client_test.go file does contain the all the framework necessary for building a matching server in go, although it also contains code to test code intermingled in. This week I'm going to update the PR to add a test or two and hopefully see if I can get the attention of the admins.

[ig0rsky]

hey @dbilling, any updates on your progress?