lumjjb commented 3 years ago

Ocicrypt Release 1.1.0

This new release of ocicrypt contains new usable interfaces and thus is a new minor number. This issue details the notable features added as well as a checklist before making a new release that will be used by consuming projects. Target date for completion is 24th January.

Changes

There are several significant changes in release 1.1.0, this includes:

Experimental PKCS11 support enabled via ENV variable
Custom keyprovider support enabled via ENV variable

Checks before release

For the features introduced, consumers of the new feature should test to see if the latest master branch is working and if there are any complications introduced through the interaction of the 2 new features.

[x] Author manual checks for feature (Require sign off by at least one author)
- [X] pkcs11 support (@stefanberger @Gsealy)
- [x] custom keyprovider support (@pravinrajr9)
[x] Check if instructions in docs can be reproduced by following instructions provided (Require at least 1 tester each)
- [X] pkcs11 support (@stefanberger)
- [X] custom keyprovider support (@stefanberger)
[x] Check upstream usage with latest commit
- [x] cri-o (@lumjjb) https://github.com/cri-o/cri-o/pull/4457 (Tests passed but platform specific integration tests are not running - ci problem, accepting for now due to no future date where it will be available)
- [x] skopeo (@lumjjb) https://github.com/containers/skopeo/pull/1148/
- [x] buildah (@lumjjb) https://github.com/containers/buildah/pull/2892
- [X] containerd + imgcrypt (@stefanberger)

stefanberger commented 3 years ago

The following issue was due to an existing installation of containerd 1.4.3 on the system and the compiled version of 1.4.2 for example cannot properly talk to the already-installed shim from 1.4.3.

For the record: containerd+imgcrypt works with today's master as well as v1.4.3 but does not work with prior versions (v1.4.0 ... v1.4.2) for some reason. It works with 1.3.9 but not with 1.3.8 ... 1.3.5 (prior ones I did not try). I get this error here when running the imgcrypt test:

Testing creation of container from encrypted image
./script/tests/test_encryption.sh: line 355: warning: command substitution: ignored null byte in input
Error: Should have been able to run a container from encrypted image when passing keys
ctr: dial unix /run/containerd/s/b758c5212b34ef3882ebabafa611d62c8f365ce7a2dd117d22e5c85f64e66913: connect: connection refused: unknown

It has something to do with unix sockets, at least that's what made the difference when bisecting between 1.4.3 and 1.4.2.

This is the line that fails and it's not clear why 'run' fails while 'create' works (for example):

       MSG=$(sudo bash -c "$CTR run \
                --gpg-homedir ${GPGHOMEDIR} \
                --gpg-version 2 \
                --key <(echo "${GPGTESTKEY1}" | base64 -d) \
                --rm \
                ${ALPINE_ENC} testcontainer1 echo 'Hello world'" 2>&1)
        failExit $? "Should have been able to run a container from encrypted image when passing keys\n${MSG}"

This is the only test that fails. When deactivating that line all other tests pass (with probably all the other versions above).

stefanberger commented 3 years ago

Containerd did not support passing environment variables to a StreamProcessor in 1.4.3. This prevents us from passing environment variables related to ocicrypt config files (OCICRYPT_CONFIG=internal, OCICRYPT_KEYPROVIDER_CONFIG=...). Instead, containerd would have to be started in an environment where these variables are already set, or we have to put the config files at default that ocicrypt uses if the variables are not set.

This PR https://github.com/containerd/containerd/pull/4916 fixed this and this feature will only become available with 1.5.

lumjjb commented 3 years ago

For the record: containerd+imgcrypt works with today's master as well as v1.4.3 but does not work with prior versions (v1.4.0 ... v1.4.2) for some reason. It works with 1.3.9 but not with 1.3.8 ... 1.3.5 (prior ones I did not try). I get this error here when running the imgcrypt test:

I can't seem to reproduce this though. Seems to work fine for me.

vagrant@ubuntu-focal:~/go/src/github.com/containerd/imgcrypt$ /home/vagrant/go/src/github.com/containerd/containerd/bin/containerd --version
containerd github.com/containerd/containerd v1.3.8 7fb6e171309113ddcb8ea9599e34321550469250
vagrant@ubuntu-focal:~/go/src/github.com/containerd/imgcrypt$ CONTAINERD=/home/vagrant/go/src/github.com/containerd/containerd/bin/containerd ./script/tests/test_encryption.sh 2>&1 >  /dev/null
bash: line 1: 22554 Killed                  /home/vagrant/go/src/github.com/containerd/containerd/bin/containerd -c /tmp/tmp.ErSygiFdZG/config.toml &> /tmp/tmp.ErSygiFdZG/log
bash: line 1: 24946 Killed                  /home/vagrant/go/src/github.com/containerd/containerd/bin/containerd -c /tmp/tmp.8wPkhfXBoB/config.toml &> /tmp/tmp.8wPkhfXBoB/log
kill: (24946): No such process
vagrant@ubuntu-focal:~/go/src/github.com/containerd/imgcrypt$ echo $?
0

pravinrajr9 commented 3 years ago

@lumjjb I have validated the key provider protocol using ocicrypt, skopoe and crio master branches, for the following scenario.

encryption using skopeo with keyprovider as binary executable: PASS decryption using skopeo with keyprovider as binary executable: PASS

encryption using skopeo with keyprovider as grpc service: PASS decryption using skopeo with keyprovider as grpc service: PASS

encryption using skopeo with keyprovider as grpc service: PASS decryption using crio with keyprovider as grpc service: PASS

encryption using skopeo with keyprovider as binary executable: PASS decryption using crio with keyprovider as binary executable: PASS

Could you please tell me how to mark in above checks that you mentioned?

lumjjb commented 3 years ago

Could you please tell me how to mark in above checks that you mentioned?

I will mark them. Do you mind trying the keyprovider with buildah as well?

pravinrajr9 commented 3 years ago

@lumjjb Sure, will do it, is this feature newly introduced in buildah, can you provide sample commands to encrypt and decrypt images or it is same as in skopeo

stefanberger commented 3 years ago

For the record: containerd+imgcrypt works with today's master as well as v1.4.3 but does not work with prior versions (v1.4.0 ... v1.4.2) for some reason. It works with 1.3.9 but not with 1.3.8 ... 1.3.5 (prior ones I did not try). I get this error here when running the imgcrypt test:

I can't seem to reproduce this though. Seems to work fine for me.

The reason seems to be that I had containerd 1.4.3 installed on my system and it seems to pick up the containerd-shim from that version but the protocol has changed, so 1.4.2 containerd cannot talk to the 1.4.3 shim.

lumjjb commented 3 years ago

@pravinrajr9 it's pretty much same as skopeo

lumjjb commented 3 years ago

The cri-o CI is half green, half pending, seems to be stuck or something, the tests have passed on their CI, just the testing on specific platforms are pending. Let's give this until the end of the week and if the CI still has issues, I think we have a positive enough indication that things are working

pravinrajr9 commented 3 years ago

@lumjjb I tried to build the buildah, but failed to build as it didnt have some of the repos like github.com/miekg, github.com/stefanberger, latest version vendor/google.golang.org/grpc along with vendor/golang.org/x/net/. After copying these repos and updating few(grpc, golang.org/x/net) to latest versions. I was able to build. But I am getting following error during encryption. OCICRYPT_KEYPROVIDER_CONFIG=/etc/ocicrypt-bin.json buildah copy --encryption-key provider:isecl:any oci:simple-bin:bin oci:simple-bin-enc:bin-enc unknown flag: --encryption-key

lumjjb commented 3 years ago

Hi @pravinrajr9 ,for buildah i think it's the push command, can you try that:

https://github.com/containers/buildah/blob/master/docs/buildah-push.md

pravinrajr9 commented 3 years ago

buildah push with encryption and pull with decryption is working. But After pushing encrypted image using buildah push, when I do crictl pull using cri-o, i get some error, FATA[0000] pulling image failed: rpc error: code = Unknown desc = invalid reference format I am not sure, whether this is expected,

pravinrajr9 commented 3 years ago

Seems my docker registry, is messed up. I will re check

lumjjb commented 3 years ago

Sounds good, once that's good we will make the release!

pravinrajr9 commented 3 years ago

Validated with buildah

encryption using buildah push with keyprovider as binary executable: PASS decryption using buildah pull with keyprovider as binary executable: PASS

encryption using buildah push with keyprovider as grpc service: PASS decryption using buildah pull with keyprovider as grpc service: PASS

encryption using buildah push with keyprovider as grpc service: PASS decryption using crio with keyprovider as grpc service: PASS

encryption using buildah push with keyprovider as binary executable: PASS decryption using crio with keyprovider as binary executable: PASS

lumjjb commented 3 years ago

Hehe done just about on schedule! Released https://github.com/containers/ocicrypt/releases/tag/v1.1.0!

containers / ocicrypt

ocicrypt 1.1.0 release checks #39

Ocicrypt Release 1.1.0

Changes

Checks before release