How Actually the Encryption Works?

[auzty]

I try to understand about encryption image layer, how this actually works?

I successfully encrypt the images, export it to file, and then import it to other machine,

and then I import it using ctr-enc it's working and the encrypted images loaded to image layer,

but when I run it with standard ctr (get from apt), it can run the images even I didn't specify the key and didn't put any key to /etc/containerd/ocicrypt/keys / ~/.config/containerd/ocicrypt/keys

my assume -> need ctr-enc to run the encrypted images

but the actual -> you can run encrypted images using standard ctr (after import / pull it using ctr-enc)

[stefanberger]

I successfully encrypt the images, export it to file, and then import it to other machine,

and then I import it using ctr-enc it's working and the encrypted images loaded to image layer,

but when I run it with standard ctr (get from apt), it can run the images even I didn't specify the key and didn't put any key to /etc/containerd/ocicrypt/keys / ~/.config/containerd/ocicrypt/keys

my assume -> need ctr-enc to run the encrypted images

You need ctr-enc to encrypt and decrypt the image. Once you have decrypted it you could also use ctr to run the image.

but the actual -> you can run encrypted images using standard ctr (after import / pull it using ctr-enc)

Are you sure the image you are running was actually encrypted? ctr does not have support for decrypting an image and an encrypted image would have to go through the decryption step before running.

[auzty]

root@podman:~# $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
   #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
   0   sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99   linux/amd64   3623844                          
   1   sha256:78c48dda0d5c416095b8f9789e306954ec87c158cd718bf90eea20872ac2e0f9   linux/amd64   2692511                          
   2   sha256:cb257a418ffad6ecd8585073bda85f359ff9a4cacbcd107341fbf14c7eb0464c   linux/amd64       336                          
root@podman:~# $CTR images layerinfo --platform linux/amd64 bash.enc:latest
   #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
   0   sha256:1cd2444771d68c8cc3f1910bce12aebc26bbe7e2d9e7b521ff60aba3c12d279f   linux/amd64   3623844          jwe        [jwe]
   1   sha256:e2e6b476940a2741dcf3ebb8a69120825e51c69358c7c87b043336407d727f7a   linux/amd64   2692511          jwe        [jwe]
   2   sha256:09a5cd6253c9140f4a7cfe3a788cf3203bd2e70013cf93ef0e3bd00b5ec14401   linux/amd64       336          jwe        [jwe]
root@podman:~# $CTR run --rm bash.enc:latest test echo 'Hello World!'
ctr: you are not authorized to use this image: missing private key needed for decryption:

root@podman:~# $CTR run --rm --key mykey.pem bash.enc:latest test echo 'Hello World!'
Hello World!
root@podman:~# ctr -a /tmp/run/containerd/containerd.sock  run --rm bash.enc:latest test echo 'Hello World!'
Hello World!
root@podman:~#

this is my config.toml

version = 2
disabled_plugins = ["io.containerd.grpc.v1.cri"]
root = "/tmp/var/lib/containerd"
state = "/tmp/run/containerd"
[grpc]
  address = "/tmp/run/containerd/containerd.sock"
  uid = 0
  gid = 0

[stream_processors]
    [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
        returns = "application/vnd.oci.image.layer.v1.tar+gzip"
        path = "/usr/local/bin/ctd-decoder"
    [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.zstd"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+zstd+encrypted"]
        returns = "application/vnd.oci.image.layer.v1.tar+zstd"
        path = "/usr/local/bin/ctd-decoder"
    [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
        returns = "application/vnd.oci.image.layer.v1.tar"
        path = "/usr/local/bin/ctd-decoder"

idk what is missing..

[stefanberger]

root@podman:~# $CTR run --rm --key mykey.pem bash.enc:latest test echo 'Hello World!' Hello World!

In this step you are decrypting the image as part of running it. The decrypted layers will be laying around after this step.

root@podman:~# ctr -a /tmp/run/containerd/containerd.sock run --rm bash.enc:latest test echo 'Hello World!' Hello World!

In this step you are running the previously decrypted image again.

If you now clean the image layer cache thoroughly and pull the encrypted image from the repo then you will not be able to run your 2nd command (plain ctr) before you run the first one.

[stefanberger]

Here's part of a test case that also shows how to clean the cache afterwards: $CTR images rm --sync ${BASH_ENC} &>/dev/null

https://github.com/containerd/imgcrypt/blob/47481fe343bd6a44b1774c6814d87ba35be72661/script/tests/test_encryption.sh#L673-L692

[stefanberger]

Actually, the fact that decrypted layers are laying around and can be used by plain ctr has something to do with the snapshot cache:

Reproducing your issue:

> sudo $CTR -a /tmp/tmp.B7qXmD8Oco/containerd.socket run  --key /tmp/tmp.B7qXmD8Oco/mykey2.jwk --rm docker.io/library/bash.enc:latest test echo "Hello World!"
Hello World!
> sudo /usr/local/bin/ctr -a /tmp/tmp.B7qXmD8Oco/containerd.socket run  --rm docker.io/library/bash.enc:latest test echo "Hello World!"
Hello World!

Clear snapshot cache and try again with /usr/local/bin/ctr:

> $CTR snapshot ls
KEY                                                                     PARENT                                                                  KIND
sha256:819491003514b1bb52fdfbd7898633157e8a347b3758bcc833059bb303b36e86 sha256:f9fd235d2fb57a206910e740995911018420b592ae7c1678e87f5a9089a5e406 Committed
sha256:94e5f06ff8e3d4441dc3cd8b090ff38dc911bfa8ebdb0dc28395bc98f82f983f                                                                         Committed
sha256:f9fd235d2fb57a206910e740995911018420b592ae7c1678e87f5a9089a5e406 sha256:94e5f06ff8e3d4441dc3cd8b090ff38dc911bfa8ebdb0dc28395bc98f82f983f Committed

> $CTR snapshot rm sha256:819491003514b1bb52fdfbd7898633157e8a347b3758bcc833059bb303b36e86
> sudo $CTR snapshot ls
KEY PARENT KIND

> sudo /usr/local/bin/ctr -a /tmp/tmp.B7qXmD8Oco/containerd.socket run  --rm docker.io/library/bash.enc:latest test echo "Hello World!"
INFO[0000] apply failure, attempting cleanup             error="failed to extract layer sha256:94e5f06ff8e3d4441dc3cd8b090ff38dc911bfa8ebdb0dc28395bc98f82f983f: read payload: read configFd: bad file descriptor\n: unknown" key="extract-651862197-UMG3 sha256:94e5f06ff8e3d4441dc3cd8b090ff38dc911bfa8ebdb0dc28395bc98f82f983f"
ctr: failed to extract layer sha256:94e5f06ff8e3d4441dc3cd8b090ff38dc911bfa8ebdb0dc28395bc98f82f983f: read payload: read configFd: bad file descriptor

$CTR snapshot ls showed sha256:94e5f06ff8e3d4441dc3cd8b090ff38dc911bfa8ebdb0dc28395bc98f82f983f that we just deleted. /usr/local/bin/ctr thinks this layer should be around but it isn't anymore after the deletion. So there's some caching going on.