support HSMs for signing images

[Hendrik-H]

/kind feature

Description Currently it is possible to sign images using OpenPGP. While OpenPGP is supported by security keys like YubiKey (https://www.yubico.com/products/) or Nitrokey (https://www.nitrokey.com/#comparison) there is usually no support by HSMs like:

• https://www.yubico.com/product/yubihsm-2
• https://www.nitrokey.com/#comparison
• https://www.ibm.com/security/cryptocards/pciecc/overview

I'm especially interested in the support of IBM crypto card as it is on FIPS PUB 140-2 level 4.

All of those cards offer PKCS#11 and https://github.com/alonbl/gnupg-pkcs11-scd might be able to be used to link those two worlds but the support seems to be limited and there does not seem to be much happening in that project. So it would be nicer if the podman ecosystem would have a more direct support for HSMs.

The easiest solution seems to be to support OpenSSL as that supports PKCS#11 engine plugins (see https://github.com/OpenSC/libp11). This Go OpenSSL library states that is is a wrapper around the native library and thus I assume it is able to use those plugins as well: https://godoc.org/github.com/spacemonkeygo/openssl

For the signing I see at least two options using OpenSSL:

1. Use CMS (Cryptographic Message Syntax): This is somewhat similar to what is done with OpenPGP. One could sign the image manifest as it is done for OpenPGP and the signature would include the JSON. From the command line the signing looks like this: openssl cms -sign -signer cert.pem -in test.data -inkey key.pem -nodetach -binary -outform pem -out test.signature. And the verification like this: openssl cms -verify -CAfile cert.pem -in test.signature -inform pem.
2. Use a digest: This way the signature does not include the original data anymore. Currently the code seems to read out the JSON data from the signature file. But I don't think that is required as the hash of the manifest JSON is required to find the correct signature file anyway. The signing would look like this: openssl dgst -sha256 -sign key.pem -out test.signed test.data. And the verification like this: openssl dgst -sha256 -verify pubkey.pem -signature test.signed test.data.

Beside allowing the signing directly via podman or skopeo it would also be nice if it would be easier to create the signature externally to those tools. For OpenPGP it is quite easy to manually create the signature file that can then be used by podman to verify the image. The only tricky part is to create the manifest.json file, which is the input for the signing. I have not found any documentation (besides the source code) on that and creating it manually one needs to use the same white space as podman does. It would be nice if there was a way to get this data for an image so that one can create the signature externally. The advantage of this is that one can have a well secured signing server that does not need to have much software installed nor requires the full image to be transferred to it. It would be enough to send the image digest or a small JSON file to this specially secured signing server and then do the signing there directly. For that it would also be nice if the signature store directory was better documented. I have not found anything about it and it is pretty easy so far but in case it was supported to provide the signature directly some more documentation would be good. An option could also be to have a command to import the signature. In that case the store would still be internal, which would be better I believe.

[fatherlinux]

@mtrmac PTAL. Thoughts? :-)

[mtrmac]

/kind feature Description All of those cards offer PKCS#11 and https://github.com/alonbl/gnupg-pkcs11-scd might be able to be used to link those two worlds but the support seems to be limited and there does not seem to be much happening in that project. So it would be nicer if the podman ecosystem would have a more direct support for HSMs.

I imagine this would use something closer to the X.509 ecosystem instead of continuing to use the OpenPGP format. The c/image policy config format anticipates that to be possible, but no code to use PKCS#11 nor a non-OpenPGP format exists currently.

For HSMs, a major concern is compatibility and testing; PKCS#11 is more or less a “do anything” interface and it’s not so obviously and trivially true that any PKCS#11 client can successfully work with any PKCS#11 server, that this could be built without targeting specific hardware, having that hardware available, and likely establishing a long-term testing process against that hardware. (Or maybe this is all fine and I’m just unfamiliar.)

Secondarily, we would have to figure out FIPS-140 certification of the code; the current “shell out to GnuPG” approach allows us to use the RHEL FIPS-140 certified module. I don’t know at all whether that is practical to do with OpenSSL (Go and FIPS-140 zeroization may not mix well), and we would necessarily have to provide a Go-only verification path in addition to the HSM implementation.

1. Use CMS (Cryptographic Message Syntax): This is somewhat similar to what is done with OpenPGP. One could sign the image manifest as it is done for OpenPGP and the signature would include the JSON.

Yes, something like that would probably be the approach. Otherwise matching the signature JSON and the signature object would introduce extra complexity.

---

Beside allowing the signing directly via podman or skopeo it would also be nice if it would be easier to create the signature externally to those tools. For OpenPGP it is quite easy to manually create the signature file that can then be used by podman to verify the image. The only tricky part is to create the manifest.json file, which is the input for the signing. I have not found any documentation (besides the source code) on that and creating it manually one needs to use the same white space as podman does.

White space is the last of the problems; you need to know how podman will compress the layers on push!

Just don’t do that; push the image to a temporary place, download the resulting EDITdigestmanifest, and sign that, if you have to do it completely independently. But I’d recommend instead replacing /usr/bin/gnupg* with a script that connects to your server, keeping the signing process integrated in the push (in particular this prevents the registry from maliciously modifying the image between the initial upload and signing).

The advantage of this is that one can have a well secured signing server that does not need to have much software installed nor requires the full image to be transferred to it.

(Such a server has no idea whether it is signing the right thing or a malicious backdoored image. It may still be the right design for a specific application, just a concern to be aware of.)

It would be enough to send the image digest or a small JSON file to this specially secured signing server and then do the signing there directly. For that it would also be nice if the signature store directory was better documented.

https://github.com/containers/image/tree/master/docs ?

[mtrmac]

Overall: Yes, HSM support is a legitimate feature request, but it’s a very non-trivial amount of work and ongoing maintenance.

[Hendrik-H]

@mtrmac thanks, I hadn't found that documentation before. I don't quite understand this point:

you need to know how podman will compress the layers on push!

The signature contains what is described here: https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format That digest in there is a bit complicated to determine so it's easier to just do an OpenPGP signing of the image with any key. The folder that the signature is stored in contains the key or one can of course also just verify the signature to get the full JSON. Once one has the JSON one can create a new signature with a different key anywhere.

A signing server would also not be able to check if the image you transfer to it is not a backdoor unless you do all kinds of security scanning on it. You normally want to keep the attack vector of the signing server as minimal as possible. But yes, you are right there needs to be something in place that checks that only valid images are being signed.

Regarding

Otherwise matching the signature JSON and the signature object would introduce extra complexity.

Isn't all the code needs to do verify that the manifest digest is signed? For that the openssl dgst approach should also work. Or is there anything else included in the JSON that the code would need from the signature?

[mtrmac]

you need to know how podman will compress the layers on push!

The signature contains what is described here: https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format That digest in there is a bit complicated to determine so it's easier to just do an OpenPGP signing of the image with any key.

I don’t understand at all. What aspect of “the image” is signed in this process? Easier to do what?

The folder that the signature is stored in contains the key

With the simple signing signatures, no keys of any kind are stored alongside signatures by c/image, podman, or the like, AFAIK. It almost seems as if you meant “digest” here.

or one can of course also just verify the signature to get the full JSON. Once one has the JSON one can create a new signature with a different key anywhere.

If this is talking about the https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format format, sure, given a signature, one can create a new signature with the same JSON…

But that implies that there already is a full manifest with the right contents that was signed first. It’s not sustainable to try to generate a “manifest that Podman is likely to generate” during a signing process, and the compression which affects layer digests is a bigger obstacle than spaces. (To be explicit, neither is a stable API. Have Podman generate the manifest and then use it.)

---

A signing server would also not be able to check if the image you transfer to it is not a backdoor unless you do all kinds of security scanning on it.

The way I think about it, the only way to have “trust” in signing is to fairly strongly integrate it with the build process, so that “the only practically likely way” to get an image signed is to submit a job to an automated build system (which records, for audit purposes, all inputs). But, sure, that’s just a parenthetical remark.

---

Otherwise matching the signature JSON and the signature object would introduce extra complexity.

Isn't all the code needs to do verify that the manifest digest is signed? For that the openssl dgst approach should also work. Or is there anything else included in the JSON that the code would need from the signature?

The “identity” recorded in the JSON is essential to prevent image substitution attacks.

[mtrmac]

(BTW Skopeo provides several subcommands that can be helpful for creating or reviewing signatures without the full Podman integration, including something to help with the manifest digest value. In general I’d fairly strongly recommend defaulting to the Podman pull/push and Skopeo copy functionality; the specialized subcommands may be useful for separately-implemented HSM integrations, if you already understand the full format and all the concerns.)

[Hendrik-H]

you need to know how podman will compress the layers on push!

The signature contains what is described here: https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format That digest in there is a bit complicated to determine so it's easier to just do an OpenPGP signing of the image with any key.

I don’t understand at all. What aspect of “the image” is signed in this process? Easier to do what?

If I want to create the signing file on my own without using podman or skopeo I need to have the image manifest JSON file, which looks like this one: https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format I have not found any easy way to get that file. That's why I also don't see how I could use https://github.com/containers/skopeo/blob/master/docs/skopeo-manifest-digest.1.md, which otherwise looks interesting. I believe I was not able to find a way to get the docker-manifest-digest. Instead of generating the file one can however also just sign the container image with skopeo and then decrypt the generated file with OpemPGP, which will give you the JSON file.

The folder that the signature is stored in contains the key

With the simple signing signatures, no keys of any kind are stored alongside signatures by c/image, podman, or the like, AFAIK. It almost seems as if you meant “digest” here.

I guess the confusion is caused a bit by skopeo / podman calling the file that results from the signing process signature-1. I was referring to that file.

or one can of course also just verify the signature to get the full JSON. Once one has the JSON one can create a new signature with a different key anywhere.

If this is talking about the https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format format, sure, given a signature, one can create a new signature with the same JSON…

But that implies that there already is a full manifest with the right contents that was signed first. It’s not sustainable to try to generate a “manifest that Podman is likely to generate” during a signing process, and the compression which affects layer digests is a bigger obstacle than spaces. (To be explicit, neither is a stable API. Have Podman generate the manifest and then use it.)

I still don't quite see what the layers matter here. Is it because the layer compression would effect the final digest of the image and thus the docker-manifest-digest? Anyhow, I believe we agree that external code should not try to generate that manifest JSON. But how would I get it if I want to perform the OpenPGP signing on my own? The only somewhat clean option seems to sign the image with some key and then decrypt the signature-1 file.

A signing server would also not be able to check if the image you transfer to it is not a backdoor unless you do all kinds of security scanning on it.

The way I think about it, the only way to have “trust” in signing is to fairly strongly integrate it with the build process, so that “the only practically likely way” to get an image signed is to submit a job to an automated build system (which records, for audit purposes, all inputs). But, sure, that’s just a parenthetical remark.

Otherwise matching the signature JSON and the signature object would introduce extra complexity.

Isn't all the code needs to do verify that the manifest digest is signed? For that the openssl dgst approach should also work. Or is there anything else included in the JSON that the code would need from the signature?

The “identity” recorded in the JSON is essential to prevent image substitution attacks.

Ah, ok, that is true. In case one references the image by digest this should not matter. But that's not done that often.

[Hendrik-H]

(BTW Skopeo provides several subcommands that can be helpful for creating or reviewing signatures without the full Podman integration, including something to help with the manifest digest value. In general I’d fairly strongly recommend defaulting to the Podman pull/push and Skopeo copy functionality; the specialized subcommands may be useful for separately-implemented HSM integrations, if you already understand the full format and all the concerns.)

As stated in the earlier reply I'm missing a command to get the manifest JSON. As said it would be nice to be able to create the signature-1 file by myself but still have podman verify that when an image is pulled or even when a container is started. Using stand alone code I could sign the image digest and verify that before I start the container. But I would really prefer the checking to be integrated to ensure it is being performed.

[mtrmac]

If I want to create the signing file on my own without using podman or skopeo I need to have the image manifest JSON file, which looks like this one: https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format

(That’s not an “image manifest”; image manifest is the thing manifest-digest contains a digest over. This JSON embedded in the signature does not have a specific name.)

I have not found any easy way to get that file.

That’s generated as part of the signature; there isn’t an API to generate it from its components. Hence the suggestion to use (podman push) and override the GNUPG executable.

If you want to implement different code doing the signing, that’s, of course, fine, but in that case please generate it yourself, with an appropriate optional.creator value.

That's why I also don't see how I could use https://github.com/containers/skopeo/blob/master/docs/skopeo-manifest-digest.1.md, which otherwise looks interesting. I believe I was not able to find a way to get the docker-manifest-digest. Instead of generating the file one can however also just sign the container image with skopeo and then decrypt the generated file with OpemPGP, which will give you the JSON file.

The folder that the signature is stored in contains the key

With the simple signing signatures, no keys of any kind are stored alongside signatures by c/image, podman, or the like, AFAIK. It almost seems as if you meant “digest” here.

I guess the confusion is caused a bit by skopeo / podman calling the file that results from the signing process signature-1. I was referring to that file.

Yes; I can’t think of a directory containing signature-1 (neither using the dir: transport, nor possibly in /var/lib/containers that would contain keys of any kind. (The signature itself only IIRC contains a key fingerprint, not a key object.)

or one can of course also just verify the signature to get the full JSON. Once one has the JSON one can create a new signature with a different key anywhere.

If this is talking about the https://github.com/containers/image/blob/master/docs/containers-signature.5.md#json-data-format format, sure, given a signature, one can create a new signature with the same JSON… But that implies that there already is a full manifest with the right contents that was signed first. It’s not sustainable to try to generate a “manifest that Podman is likely to generate” during a signing process, and the compression which affects layer digests is a bigger obstacle than spaces. (To be explicit, neither is a stable API. Have Podman generate the manifest and then use it.)

I still don't quite see what the layers matter here. Is it because the layer compression would effect the final digest of the image and thus the docker-manifest-digest?

Yes. An image manifest can only be “observed” for an already-compressed image (i.e. image that already is on a registry, or is in the process of being pushed); it doesn’t make sense to create it independently because the implementation pushing the image compresses it in an unknown way, and formats the manifest JSON in an unknown way, and both can change at any time for any reason.

(One way to do that is skopeo copy $source dir:… and see the manifest file in there (if the image is not multi-arch). In Go, it would be simpler to use the Go API of c/image directly.)

[rhatdan]

@Hendrik-H @mtrmac Where are we this?

[Hendrik-H]

Well, I'm still interested in signing support that can leverage a HSM. In the mean time I'm looking at signing images out of band. So sign and verify image digests outside of podman.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mtrmac should we move this up in priority?

[mtrmac]

For HSMs, a major concern is compatibility and testing

but it’s a very non-trivial amount of work and ongoing maintenance

Basically this needs to be corporate-prioritized and -budgeted, by some corporation with the appropriate hardware and interested in at least watching over the relevant repos to notice accidental breakage.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This either needs volunteers to work on it, or we need guidance on priorities and funding from Red Hat to work on it.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[fatherlinux]

@mtrmac can't we just test and verify with a standard PKCS#11 interface and let the hardware vendors meet us in the middle? Maybe I misunderstand the PKCS#11 API? I'll fully admit, I always thought PKCS#11 was it's own key/cert format from my days managing Microsoft Web Servers, but I took a look at this: https://support.nitrokey.com/t/nitrokeypro2-opensc-vs-pkcs11-vs-openpgp-card-vs-gpg/2497/2

Second question, could that Cosign project help with this interface between GPG simple keys and PKCS#11?

[mtrmac]

@mtrmac can't we just test and verify with a standard PKCS#11 interface and let the hardware vendors meet us in the middle?

Shouldn’t I be asking you that? :)

There is one way that should be quite practical: Find a vendor that declares to be a drop-in pluggable into, or a drop-in replacement for, GnuPG. Ultimately the c/image implementation just calls /usr/bin/gpg* to create a signature, or verify one, with no parameters. So if the hardware can be just used as a GPG backend, any necessary GPG configuration can be done by the user using GPG configuration or the like, and this is something the vendor can probably support without any specific support by c/image. That’s the YubiKey example above, but also insufficient for the original reporter.

---

For other ways, e.g. to integrate with an existing X.509 ecosystem, if we can’t trivially farm that out to a guaranteed-supported caller with its own configuration (which I don’t think we can, but I could be wrong), I don’t think this can be done without continued testing:

PKCS#11 is an ABI to invoke cryptographic operations. It doesn’t, IIRC, prescribe that any particular operation will actually be possible — you could use PKCS#11 to talk to a single-purpose RSA hardware (which could work for signatures), or single-purpose AES hardware (which is useless).

We can certainly write and test software against SoftHSM or something like that, and that might be good enough for an upstream effort. The concerns are:

• Suppose someone reads that “PKCS#11 is supported”, buys an expensive HSM, and it just doesn’t work because no-one ever tried it with that machine and its limitations. What happens in the commercial relationships (any of user-HW vendor, user-software vendor, HW vendor-software vendor)? Who is going to bear the cost of the machine, or the software updates to make that work?
• Suppose everything works fine, and then we make a code change that is thoroughly tested and works perfectly against that SoftHSM, but happens to break the hardware deployment. What happens in the commercial relationships?

That’s why I think

Basically this needs to be corporate-prioritized and -budgeted, by some corporation with the appropriate hardware and interested in at least watching over the relevant repos to notice accidental breakage.

so that the breakage, if any, is not first discovered by the customer who has already paid.

---

Maybe I misunderstand the PKCS#11 API? I'll fully admit, I always thought PKCS#11 was it's own key/cert format from my days managing Microsoft Web Servers

IIRC it’s much more low-level than certificates, basically it only performs the raw cryptographic algorithm, and any data formatting and crypto systems around that are up to callers. (See e.g. https://docs.oasis-open.org/pkcs11/pkcs11-curr/v3.0/os/pkcs11-curr-v3.0-os.html ).

But we also need to somehow integrate with the rest of the organization that is probably using the same HSM or a set for other uses, and only ask the HSM to perform operations that it actually implements. My impression is that those things all lean towards the X.509 formats, not OpenPGP (and some searches suggest that GnuPG intentionally does not support the PKCS#11 interface at all, and actual uses require a shim) — but I could well be mistaken.

---

Second question, could that Cosign project help with this interface between GPG simple keys and PKCS#11?

Right now, it would make that more difficult, because an explicit goal is to be Go-native, but PKCS#11 is a C-based ABI. I guess that this goal could be changed, but I don’t know.

The concerns about supportability would probably not change.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[DemiMarie]

I think the simplest approach would be to send the raw bytes that are to be signed to a subprocess over stdin, and wait for the subprocess to respond with a signature on stdout. The subprocess would take a key fingerprint as its only argument, and would be responsible for the entire signing operation. Anyone can then make a binary that implements this API. For instance, one could write a C program that used PKCS#11, or a shell script that wrapped a Qubes OS qrexec service.

[mtrmac]

That’s already possible; the signing implementation calls an external GPG binary.

[DemiMarie]

@mtrmac Parsing the GPG command line is non-trivial; Qubes OS Split GPG gives an idea of the complexity involved. That’s why I suggested the calling convention I did: it is trivial to convert it to the calling convention GPG uses, but the reverse is not true.

[countofsanfrancisco]

I think we need to simply the HSM compatibility and testing problem. HSMs are varied and there really isn't any standard protocol for it. The closes one is the KMIP protocol. One mentioned CMS earlier. Even these two protocols are not widely implemented for HSM. I think a lot of HSMs are proprietary so asking to support HSM may not be the best way to support HSM. Perhaps, the best way to support HSM to support the ability where the keys do not exist on the same machine as Open PGP or podman or gpg exists.

One way is that podman provides us a "callback" API or something equivalent to OpenSSL engines, where the APIs are predefined. Or, better yet, just as the original poster of this issue stated just link to OpenSSL and call their signing APIs. This allows us to write our layer of code to communicate to our HSM and give back the signature to podman, gpg (or whatever) in the format podman, gpg. From a compatability standpoint, podman only needs to support the API callback (or openssl in the later scenario) and its desired behavior. The administrators of podman in the datacenter will deal with integrating this into their HSM via the APIs (or openssl engine). In this method, there is a clear delineration between what the podman/openpgp/gpg needs to test and support and what we are responsible for. Otherwise, you would be blocked by the testing of HSM and technological issues that I don't think should be podman's responsibility. I just need these "hookpoints" so that I can write my own layer to interface with my HSM in the way my HSM's infrastructure wants me to. Also, the hookpoints should allow us to prompt for additional information or have some of podman's command line parameters passed to the "hooked" code.

[Hendrik-H]

The callback approach sounds good to me. In a addition to the actual signing and verification it should also possible to somehow easily create the hash for the image. I for example need to sign SHA-3 hashes and not SHA-2 hashes.

[rhatdan]

@mtrmac Thoughts?

[mtrmac]

The “raw bytes to sign” are currently constructed, and encapsulated in a message format, by GPG, not directly under control of Go code. That’s why replacing the /usr/bin/gpg* binary is an obvious interaction point.

Defining some other plugin interface is plausible in principle, but we are talking about:

• a completely new codepath to construct GPG payloads
• a completely new plugin interface
• that we wouldn’t support or use (or even test much?)

That’s not trivial. It’s not a feature that already exists and we just stubbornly refuse to expose. On the Go level, I suppose somehow combining golang.org/x/crypto/openpgp with a custom crypto.Signer would be the approach — while somehow strongly preventing/discouraging users from using OpenPGP with in-memory Go-implemented keys, which are incompatible with FIPS requirements.

(Cosign would affect the “construct a payload / message” part, because it essentially already does those things, at least.)

[DemiMarie]

The “raw bytes to sign” are currently constructed, and encapsulated in a message format, by GPG, not directly under control of Go code. That’s why replacing the /usr/bin/gpg* binary is an obvious interaction point.

As someone who has worked on a program (Qubes OS split-gpg v1) that tries to do exactly that, I strongly recommend taking a different route.

The problem with replacing /usr/bin/gpg* is that the GPG command-line interface is very difficult to reimplement outside of GPG. My gpg has 422 command-line options, some of which take arguments and some of which do not. As of QubesOS/qubes-app-linux-split-gpg@bd3b0bf0e8636bd95c485e7f70b70fca4d080222, the C code in split-gpg understands 109 long options and 16 short options, and passes through 98 of them. There is also a bash wrapper script that handles 37 options the C code does not know about, and treats 37 others specially even though the C code does know about them. This has proven to be an absolute nightmare to maintain, and I implore you to not ask others to implement something similar.

A vastly better approach is to expect the signing program to expose a very simple API that is independent of GPG. While this API would not be implemented by GPG itself, it would be trivial to write a shell script based on GPG that did implement it. The API could be as simple as having the signing key fingerprint and (optional) hash algorithm as the only arguments, with the data to be signed on stdin and the signature returned on stdout. Something like the following should work (disclaimer: untested):

#!/bin/sh --
set -eu
if [ "$#" -lt 1 ] || [ "$#" -gt 2 ]; then
   echo "Usage: simplesign FINGERPRINT [HASH_ALGORITHM]" >&2
   exit 1
fi
exec gpg --sign --batch --no-tty --detach-sign ${2+"--digest-algo=$2"} "--local-user=$1!" -o -

You may consider this tiny script to be in the public domain or CC0 (your choice).

[mtrmac]

Sure. https://github.com/containers/image/blob/0899a2be301675ac067c3d9f0bec7c8c5bb9bb81/signature/mechanism.go#L20-L38 exists in Go; if someone wants to contribute (and maintain) a third implementation to c/image, and have that implementation shell out to whatever, that’s reasonable. But at given other work (e.g. the Cosign transition), it’s not code we are likely to write, and it’s not something we are likely to notice if it breaks.

[vrothberg]

I moved the issue to containers/image.