Error when import images from tar file

[tranducanhbk]

Hi! I alread successfully export a images file with command: sudo ctr-enc images export --platform linux/arm64 --skip-manifest-json --skip-non-distributable image.tar russian_lpr.enc:latest But when i import this tar file by command: sudo ctr-enc images import --platform linux/arm64 --skip-decrypt-auth --digests image.tar has erros like below

unpacking russian_lpr.enc:latest (sha256:7b9e5f040ab776c4ad8bd0838ebb6a4f82b730c524835156f7119cd11275da81)...INFO[0048] apply failure, attempting cleanup error="failed to extract layer sha256:e996f7b5c172e7efdf80c4315098351ede3c56e08566a20be161ff253df969cc: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount654180591: archive/tar: invalid tar header: unknown" key="extract-630916527-yqAD sha256:e996f7b5c172e7efdf80c4315098351ede3c56e08566a20be161ff253df969cc" ctr: failed to extract layer sha256:e996f7b5c172e7efdf80c4315098351ede3c56e08566a20be161ff253df969cc: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount654180591: archive/tar: invalid tar header: unknown

Do u have any ideal for this issue?

[lumjjb]

it looks like it may be using the encrypted layer directly without decrypting it so it doesn't recognize the file header, quick question what's the rationale behind --skip-decrypt-auth on an encrypted image?

[tranducanhbk]

Hi lumjjb! U mean need to decypt images before unpack the images? I use --skip-decrypt-auth parameter to not decryt image when import. I only decryt images when run container.

Could u give me the right command to export and import a images. Below is command i use to export and import a images. 1) sudo ctr-enc images pull --platform linux/arm64 docker.io/tranducanh7/russian_lpr:latest 2) sudo ctr-enc images encrypt --recipient jwe:mypubkey.pem --platform linux/arm64 docker.io/tranducanh7/russian_lpr:latest russian_lpr.enc:latest 3) sudo ctr-enc images export --platform linux/arm64 --skip-manifest-json --skip-non-distributable image.tar russian_lpr.enc:latest 4) sudo ctr-enc images import --platform linux/arm64 --skip-decrypt-auth --digests image.tar

[stefanberger]

@tranducanhbk Are you using the latest version of imgcrypt?

Here's a sequence that you should try including export and import and the steps in between: https://github.com/containerd/imgcrypt/blob/main/script/tests/test_encryption.sh#L281-L303

[stefanberger]

    $CTR images import \
        --all-platforms \
        --base-name ${ALPINE_ENC_IMPORT_BASE} \
        ${WORKDIR}/${ALPINE_ENC_EXPORT_NAME} &>/dev/null
    if [ $? -eq 0 ]; then
        failExit 1 "Import of encrypted image without passing PGP key should not have succeeded"
    fi

Importing without being able to decrypt it doesn't work with ctr-enc.

[stefanberger]

I have a pending PR now that fixes the issue as far as this is possible at the moment From what I can see the ctr import --platform doesn't work properly with the containerd 1.6.8 ctr command.

Here's a baseline script just for exporting and importing a plain image:

#!/usr/bin/env bash
set -x

ALPINE=quay.io/jitesoft/alpine:latest
PLATFORM="--platform linux/amd64"

sudo ctr images pull ${PLATFORM} ${ALPINE}

rm -f image.tar
sudo ctr images export ${PLATFORM}  --skip-manifest-json --skip-non-distributable image.tar ${ALPINE}

sudo ctr images rm --sync ${ALPINE}

#CTR=ctr
#while [ -n "$(sudo $CTR snapshot ls | tail -n +2)" ]; do
#       for v in $(sudo $CTR snapshot ls | tail -n +2 | cut -d" " -f1); do
#               sudo $CTR snapshot rm $v &>/dev/null
#       done
#done

sudo ctr images import ${PLATFORM} --base-name ${ALPINE} image.tar

This script fails at the import with:

ctr: content digest sha256:51382c66150ddafb090aa5ed7a18e10740c263e0b3b4f0fd5562ab2b367c7906: not found

So we won't be able to do better with encrypted image import when using --platform.

Here's a script for encrypted image import:

#!/usr/bin/env bash
set -x

ALPINE=quay.io/jitesoft/alpine:latest
ALPINE_ENC=quay.io/jitesoft/alpine:enc
ALPINE_ENC_IMPORT_BASE=quay.io/jitesoft/alpine-import
PLATFORM="--platform linux/amd64"

sudo ctr-enc images rm --sync ${ALPINE_ENC}

sudo ctr-enc images pull ${PLATFORM} ${ALPINE}
sudo ctr-enc images layerinfo ${ALPINE}

sudo ctr-enc images encrypt ${PLATFORM} --recipient jwe:mypubkey.pem ${ALPINE} ${ALPINE_ENC}
sudo ctr-enc images layerinfo ${ALPINE_ENC}
rm -f image.tar

sudo ctr-enc images export ${PLATFORM} image.tar ${ALPINE_ENC}
sudo ctr-enc images rm --sync ${ALPINE_DEC} ${ALPINE}

#CTR=ctr-enc
#while [ -n "$(sudo $CTR snapshot ls | tail -n +2)" ]; do
#       for v in $(sudo $CTR snapshot ls | tail -n +2 | cut -d" " -f1); do
#               sudo $CTR snapshot rm $v &>/dev/null
#       done
#done
#sudo ctr-enc snapshot ls

sudo ctr-enc images import ${PLATFORM} --base-name ${ALPINE_ENC_IMPORT_BASE} --skip-decrypt-auth image.tar

This one fails for me with the PR #91 also with:

ctr: content digest sha256:51382c66150ddafb090aa5ed7a18e10740c263e0b3b4f0fd5562ab2b367c7906: not found

[stefanberger]

When setting PLATFORM="--all-platforms" then it works.

[stefanberger]

However, when in the above script the cache-clearing is enabled then this error occurs:

$ sudo ctr-enc images import --all-platforms --base-name quay.io/jitesoft/alpine-import --skip-decrypt-auth image.tar
unpacking quay.io/jitesoft/alpine:enc (sha256:7f1bf61141020d00572b6969dd7bd2372f830b7dca257cd9f3d68962af7f7fef)...INFO[0007] apply failure, attempting cleanup             error="failed to extract layer sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897: read payload: read configFd: bad file descriptor\n: unknown" key="extract-914463986-oV71 sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897"
ctr: failed to extract layer sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897: read payload: read configFd: bad file descriptor
: unknown

In this case ctd-decoder gets an invalid configFd from containerd and cannot read from the file descriptor or decrypt afterwards ... Not sure what this is.

[tranducanhbk]

Hi stefanberger! If i don’t use option —skip-decrypt-auth in the import command I can import without errors right? After import images successfully when I run it I need key or not?

[tranducanhbk]

Hi stefanberger! when we use option --skip-decrypt-auth ctr-enc should not check key to unpack this images, but as I see our error ctr-enc try to find decryption keys in provided key path or in image.tar but not found. I think when we export images the image.tar file not include keys_layer for decrypt it lead importing miss this key

Below is my newest errors unpacking quay.io/jitesoft/alpine:enc (sha256:c3bad22cc2d112f237a3e01a0545852947cfd9e0d26404e6307467b4b809eb87)...INFO[0000] apply failure, attempting cleanup error="failed to extract layer sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897: unable to get decryption keys in provided key path: lstat /etc/containerd/ocicrypt/keys: no such file or directory\n: unknown" key="extract-575014544-DW7n sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897" ctr: failed to extract layer sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897: unable to get decryption keys in provided key path: lstat /etc/containerd/ocicrypt/keys: no such file or directory : unknown

[stefanberger]

If i don’t use option —skip-decrypt-auth in the import command I can import without errors right? After import images successfully when I run it I need key or not?

From what I am seeing the import of an image requires the decryption key but the image will not be decrypted as part of the import and an extra decryption step is needed to get a decrypted image. I documented this in PR #93 now as follows:

   ctr: Document that import of encrypted image requires decryption key

    The import of an encrypted image requires the decryption key as proof
    that one posses one of the decryption keys of the image. However, the
    image will not be decrypted as part of the import. The alternative path
    that does not require the decryption key is to pull the image from a
    repository.

    The underlying reason why one needs the key is because containerd sends
    an encrypted layer to ctd-decoder which needs the decryption key for the
    decryption of that layer and for the import to succeed.

    It is not currently clear what the layer represents and why it is part
    of an exported image. The layer that is sent for the current alpine
    image is the layer with the hash ff7f8bb.. from here:

    ./blobs/sha256/8a1591...:
    {"architecture":"amd64",[...]
     {"type":"layers",
      "diff_ids":
        ["sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897",
         "sha256:c32387d564776805eb144718cd41629761e1980280c9d512df358b60f9fe6ba3"]
     }
    }

[tranducanhbk]

Thank for your quickly reply. I am looking forward your new update commiting to resolve this issue.

[lumjjb]

Also as a side note, wanted to point out that if you're wanting to use this for production use, that ctr is more of a debugging containerd binary and using the library directly or other tools like nerdctl will be better!

[stefanberger]

Thank for your quickly reply. I am looking forward your new update commiting to resolve this issue.

The update is in the main branch now. However, you will have to pass the --key option on import. This is for then general case where the cache of snapshots is clean, otherwise it may work without the --key option but this will not work when you copy the tar to a remote machine and import it there. My other recommendation would be to push the encrypted image to a repo and pull it from there on the remote machine.

[tranducanhbk]

I have just clone and rebuild lib from your update. But nothing change. The error still occour.

These are command I used.

ALPINE=quay.io/jitesoft/alpine:latest
ALPINE_ENC=quay.io/jitesoft/alpine:enc
ALPINE_ENC_IMPORT_BASE=quay.io/jitesoft/alpine-import
PLATFORM="--all-platforms"

sudo ctr-enc images rm --sync ${ALPINE_ENC}

sudo ctr-enc images pull ${PLATFORM} ${ALPINE}
sudo ctr-enc images layerinfo ${ALPINE}

sudo ctr-enc images encrypt --platform linux/amd64  --recipient jwe:mypubkey.pem ${ALPINE} ${ALPINE_ENC}
sudo ctr-enc images layerinfo ${ALPINE_ENC}
rm -f image.tar

sudo ctr-enc images export ${PLATFORM} image.tar ${ALPINE_ENC}
sudo ctr-enc images rm --sync ${ALPINE_ENC} ${ALPINE}

sudo ctr-enc images import  ${PLATFORM} --base-name ${ALPINE_ENC_IMPORT_BASE}    --key mykey.pem --skip-decrypt-auth image.tar

[stefanberger]

Your scripts works just fine for me. Are you sure that mypubkey.pem is the public key to mykey.pem?

[tranducanhbk]

yes, mypubkey.pem is the public key to mykey.pem.

my errors looks diffences from yours unpacking quay.io/jitesoft/alpine:enc (sha256:bc79a76d273662fdf0b15dc54b0ba64006354a4a069f0880fd96af73b56f41c5)...INFO[0000] apply failure, attempting cleanup error="failed to extract layer sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897: unable to get decryption keys in provided key path: lstat /etc/containerd/ocicrypt/keys: no such file or directory\n: unknown" key="extract-140766187-VUPg sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897" ctr: failed to extract layer sha256:ff7f8bbf1c81b508f82b1c59e8c2467175c0b33e58a79507f4fde8067d6f1897: unable to get decryption keys in provided key path: lstat /etc/containerd/ocicrypt/keys: no such file or directory No directory. Do have have /etc/containerd/ocicrypt/keys thi folder

[stefanberger]

Do have have /etc/containerd/ocicrypt/keys thi folder

I have no keys there but I created this directory as root. I think that will get it to work then.

[tranducanhbk]

After i created /etc/containerd/ocicrypt/keys folder it works fine. Now i will try to run this images Thank you so much

[tranducanhbk]

It works fine now. Thank you .

[stefanberger]

Good, then I let you close the issue.

[tranducanhbk]

Yes. thanks you

[tranducanhbk]

Hi stefanberger! This only work in amd64. when i try to export and import in arm64 (example with Xavier AGX) error occur like below: unpacking russian_lpr.enc:latest (sha256:7b9e5f040ab776c4ad8bd0838ebb6a4f82b730c524835156f7119cd11275da81)...INFO[0048] apply failure, attempting cleanup error="failed to extract layer sha256:e996f7b5c172e7efdf80c4315098351ede3c56e08566a20be161ff253df969cc: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount654180591: archive/tar: invalid tar header: unknown" key="extract-630916527-yqAD sha256:e996f7b5c172e7efdf80c4315098351ede3c56e08566a20be161ff253df969cc" ctr: failed to extract layer sha256:e996f7b5c172e7efdf80c4315098351ede3c56e08566a20be161ff253df969cc: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount654180591: archive/tar: invalid tar header: unknown

in arm64 arch does'nt have /etc/containerd/ folder with config.toml file.

[stefanberger]

Did you use the same script on arm64 with the latest code for imgcrypt?

[stefanberger]

I tested it with the script on ppc64 and there it worked just fine. I think you should use the script that previously worked on amd64 on arm64 as a baseline.

[tranducanhbk]

Yes, I use same script with amd64. I think ppc64 and arm64 are diffence arch so it work in ppc64 but not with arm64.

Did u try to test in jetson nano or other nvidia device?

This is script i run

ALPINE=quay.io/jitesoft/alpine:latest
ALPINE_ENC=quay.io/jitesoft/alpine:enc
ALPINE_ENC_IMPORT_BASE=quay.io/jitesoft/alpine-import
PLATFORM="--all-platforms"

sudo ctr-enc images rm --sync ${ALPINE_ENC}

sudo ctr-enc images pull ${PLATFORM} ${ALPINE}
sudo ctr-enc images layerinfo ${ALPINE}

sudo ctr-enc images encrypt --platform linux/arm64  --recipient jwe:mypubkey.pem ${ALPINE} ${ALPINE_ENC}
sudo ctr-enc images layerinfo ${ALPINE_ENC}
rm -f image.tar

sudo ctr-enc images export ${PLATFORM} image.tar ${ALPINE_ENC}
sudo ctr-enc images rm --sync ${ALPINE_ENC} ${ALPINE}

sudo ctr-enc images import  ${PLATFORM} --base-name ${ALPINE_ENC_IMPORT_BASE} --key mykey.pem --skip-decrypt-auth image.tar

and below is errror

unpacking quay.io/jitesoft/alpine:enc (sha256:3a9645caca1d224879ee37a9281b742be86775d8a80132309f20320ba78140e0)...INFO[0000] apply failure, attempting cleanup             error="failed to extract layer sha256:c580830da4c433ff0c30b55e350158ec7f02e20652b87d9b1355eac35c90a082: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount379631465: archive/tar: invalid tar header: unknown" key="extract-229755850-fPIP sha256:c580830da4c433ff0c30b55e350158ec7f02e20652b87d9b1355eac35c90a082"
ctr: failed to extract layer sha256:c580830da4c433ff0c30b55e350158ec7f02e20652b87d9b1355eac35c90a082: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount379631465: archive/tar: invalid tar header: unknown

[stefanberger]

This is the script I am using on ppc64le and it works just fine with or without the clearing of the snapshots:

#!/usr/bin/env bash

ALPINE=quay.io/jitesoft/alpine:latest
ALPINE_ENC=quay.io/jitesoft/alpine:enc
ALPINE_ENC_IMPORT_BASE=quay.io/jitesoft/alpine-import
PLATFORM="--all-platforms"

sudo ctr-enc images rm --sync ${ALPINE_ENC}

sudo ctr-enc images pull ${PLATFORM} ${ALPINE}
sudo ctr-enc images layerinfo ${ALPINE}

sudo ctr-enc images encrypt --platform linux/ppc64le --recipient jwe:mypubkey.pem ${ALPINE} ${ALPINE_ENC}
sudo ctr-enc images layerinfo ${ALPINE_ENC}
rm -f image.tar

sudo ctr-enc images export ${PLATFORM} image.tar ${ALPINE_ENC}
sudo ctr-enc images rm --sync ${ALPINE_ENC} ${ALPINE}

#CTR=ctr-enc
#while [ -n "$(sudo $CTR snapshot ls | tail -n +2)" ]; do
#       for v in $(sudo $CTR snapshot ls | tail -n +2 | cut -d" " -f1); do
#               sudo $CTR snapshot rm $v &>/dev/null
#       done
#done
#sudo ctr-enc snapshot ls

sudo ctr-enc images import ${PLATFORM} --base-name ${ALPINE_ENC_IMPORT_BASE} --skip-decrypt-auth --key myprivkey.pem image.tar

[stefanberger]

I can also modify my script to encrypt only for linux/arm64 (on ppc64le) and export and then import also works in this case then.

[tranducanhbk]

when i encrypt for linux/ppc64le it work fine but when i set encrypt for linux/arm64 the errors happend. I am test on Jetson AGX jetpach 4.4.1

success with ppc64l3

fail with arm64

[stefanberger]

Well, I cannot recreate the issue. On both amd64 and ppc64 I can encrypt for only amd64, or arm64, or ppc64le and export and import and all cases work.

I have seen the error you are showing when I omit the private key (or possibly passed the wrong private key), but otherwise I have not seen this error.

[tranducanhbk]

I mean when we run script on OS with arch is amd64 it works well. But the error with encrypt to arm64 occurs when we test on OS with arch is arm64. I will open new issues. Is it ok?

[stefanberger]

I tried to recreate your issue on ppc64le and amd64 and could not. What actually works then on arm64? Can you use --all-platforms?

[stefanberger]

Can you show me how you created the encryption keys on the ARM machine? Is it an RSA or an EC key?

[tranducanhbk]

If u have a jetson nano or other similar to it, u can got same error like me

I use RSA. Below is command i use to create keys

openssl genrsa --out mykey.pem

openssl rsa -in mykey.pem -pubout -out mypubkey.pem

[stefanberger]

Looks good.

[stefanberger]

I don't have access to a Jetson nano...

[tranducanhbk]

If u can I will open my teamview for u access to my jesson nano to check