How to successfully run tests via local docker development environment?

jfallows commented 3 years ago

I have run into some issues while following the instructions in the README on how to build locally and test in the development environment.

Building

% tools/devtool build libvtok_p11.so
...
Finished dev [unoptimized + debuginfo] target(s) in 51.51s
[p11ne devtool] Built: /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/target/debug/libvtok_p11.so

% tools/devtool build dev-image
...
[p11ne devtool] nitro-cli build-enclave failed.

The command in devtools producing this error seems to be at line 299:

    local interactive=
    [[ -t 1 ]] && interactive="-it"
    docker run --rm $interactive \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v "$EVAULT_SRC_DIR:$CTR_SRC_DIR" \
        $signing_extra_docker_args \
        --user "$(id -u):$(id -g)" \
        --group-add=$docker_gid \
        "$PARENT_CTR_IMG" \
        nitro-cli build-enclave \
            --docker-uri "$eif_tag" \
            $signing_extra_build_args \
            --output-file "$CTR_BUILD_DIR/p11ne.eif" \
            > "$EVAULT_BUILD_DIR/image-measurements.json"
    ok_or_die "nitro-cli build-enclave failed."

In my case, the actual command being executed was:

% docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock -v /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm:/p11ne.src --user 501:20 --group-add=0 p11ne-parent:3 nitro-cli build-enclave --docker-uri p11ne-eif-tmp-ctr:latest --output-file /p11ne.src/build/p11ne.eif > /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/image-measurements.json

Checking the output of the command:

% cat build/image-measurements.json 
Start building the Enclave Image...
Docker error: PullError
[ E50 ] Docker image pull error. Such error appears when trying to build an EIF file, but pulling the corresponding docker image fails. In this case, the error backtrace provides detailed informatino on the failure reason.

For more details, please visit https://docs.aws.amazon.com/enclaves/latest/user/cli-errors.html#E50

If you open a support ticket, please provide the error log found at "/var/log/nitro_enclaves/err2021-07-27T21:08:50.720127200+00:00.log"

Attempting to verify docker is working as expected on container:

% docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock -v /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm:/p11ne.src --user 501:20 --group-add=0 p11ne-parent:3 docker version
Client:
 Version:           20.10.4
 API version:       1.41
 Go version:        go1.15.8
 Git commit:        d3cb89e
 Built:             Mon Mar 29 18:54:36 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied

So looks like a permission error on the bind mount for /var/run/docker.sock.

Attempted workaround:

% docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock -v /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm:/p11ne.src --user 501:20 --group-add=0 p11ne-parent:3 sudo chmod g+w /var/run/docker.sock
% docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock -v /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm:/p11ne.src --user 501:20 --group-add=0 p11ne-parent:3 docker version
Client:
 Version:           20.10.4
 API version:       1.41
 Go version:        go1.15.8
 Git commit:        d3cb89e
 Built:             Mon Mar 29 18:54:36 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Attempt to continue with build after applying workaround:

% tools/devtool build dev-image
...
[p11ne devtool] Built: /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/p11ne.eif
[p11ne devtool] Measurements: /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/image-measurements.json

Progress!

Testing

% tools/devtool simulate-enclave
...
[p11ne devtool] Built: /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/target/debug/libvtok_p11.so
P11_KIT_SERVER_ADDRESS=unix:path=./p11kit.sock; export P11_KIT_SERVER_ADDRESS;
P11_KIT_SERVER_PID=9; export P11_KIT_SERVER_PID;
[vToken] Provisioning server is now running

Looks good, left that running and started a separate shell to continue.

% tools/devtool simulate-parent
...
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
[p11ne devtool] Built: /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/target/debug/p11ne-agent
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
[p11ne devtool] Built: /Users/jfallows/GitHub/jfallows/aws-nitro-enclaves-acm/build/target/debug/p11ne-client
[p11ne devtool] Enclave container is not runing. Use `tools/devtool simulate-enclave` to start it.

This warning about the enclave container not running is confusing because it is definitely already started. In fact, if I stop the enclave container, then this message does not print. IIUC, then the logic in the devtools script may be inverted at line 583.

    local ctr_id=$(get_running_container_id "$ENCLAVE_CTR")
    if [[ -n $ctr_id ]]; then
        say_warn "Enclave container is not runing. Use \`$0 simulate-enclave\` to start it."
    fi

perhaps should be

    local ctr_id=$(get_running_container_id "$ENCLAVE_CTR")
    if [[ -z $ctr_id ]]; then
        say_warn "Enclave container is not runing. Use \`$0 simulate-enclave\` to start it."
    fi

if we want the warning to be presented only when $ctr_id is empty.

In any case, this warning does not prevent progress, leaving the simulated parent container running.

Attempting to continue to follow the instructions from README.md from inside the simulated parent container.

[jfallows@p11ne-parent ~] openssl pkeyutl -keyform engine -engine pkcs11 -sign -inkey "pkcs11:model=p11ne-token;manufacturer=Amazon;serial=EVT00;token=my-token-label;id=%52;type=private" -in hello.txt -out test.sig
Failed to enumerate slots
can't use that engine
140661936871328:error:260B806D:engine routines:ENGINE_TABLE_REGISTER:init failed:eng_table.c:175:
no engine specified
unable to load Private Key
Error initializing context

This seemed to indicate that the token has not been provisioned yet.

[jfallows@p11ne-parent ~] p11ne-cli describe-device
{
  "Ok": {
    "DeviceDescription": {
      "free_slot_count": 8,
      "tokens": []
    }
  }
}

Confirmed, no tokens, but simulated enclave is reachable from simulated parent.

Attempted to continue with running tests per README.md.

The tests directory contains integration tests that can be executed to validate the PKCS#11 module functionality using openssl or OpenSC pkcs11-tool. Tests can be executed via:

./tests/testtool openssl The above test suite is also applicable when using real enclaves.

This description implies being applicable for both development enclaves and real enclaves, and references the tests directory from repo, so attempted to run that from my OSX host environment.

% ./tests/testtool openssl
./tests/testtool: line 716: p11ne-cli: command not found
[testtool] Cannot stop the p11ne enclave

Realized this script does not implicitly run a docker container, so should not be executed from host environment.

Instead, a comment at the top of ./tests/testtool says:

# Test tool. Used for smoke-testing an p11ne enclave. # Right now user should copy the following on an EC2 instance # - the tests directory from the p11ne repo # - the testhelpers binary in the tests directory (pre-built in the p11ne repo) # /home/ec2-user/tests/testtool # /home/ec2-user/tests/testhelpers # ... # # With a role attached and a KMS key created beforehand run ./testtool openssl [...]

Due to the mention of EC2, it is now unclear if these tests can actually be run against a development enclave using the local docker setup.

Copying the tests directory onto the simulated parent and running testtool from there gives:

[jfallows@p11ne-parent ~] ./tests/testtool openssl
[p11ne-cli] You are running in an emulated dev container. You can use `devtool simulate-enclave` on the host machine to start up an enclave-emulating dev container.
[testtool] Cannot stop the p11ne enclave

Would appreciate some guidance on how to run the openssl tests using local docker development environment please.

alcioa commented 3 years ago

Hi again and thanks for having a look into this.

Indeed, as the script doc states, this can be executed only on an EC2 instance where you have an ACM enclave (production or debug mode) and setup accordingly. Unfortunately, in the simulated environment we did not mock the KMS decrypt path when you provision the token in the codebase enclave-side. That is, when you provision a token, it will try to access KMS endpoint based on KMS key ID and region (i.e. https://github.com/aws/aws-nitro-enclaves-acm/blob/main/src/vtok_srv/src/worker.rs#L74) The test suite is executed against a provisioned token with the keys present in the test directory. Thus you could only run it on your instance.

[ec2-user@<instance-ip>~]$ ./tests/testtool openssl --kms-key-id <your-kms-key-id> --kms-region <your-kms-region>
Running Sign&verify tests
Running RSA-PSS Sign&Verify tests using openssl
test_openssl_digestsignverify_rsapss_rsa1024_0B_0_sha1
test_openssl_digestsignverify_rsapss_rsa1024_0B_0_sha1 ............... PASSED
test_openssl_digestsignverify_rsapss_rsa1024_0B_1_sha1
...
Running RSA RAW Encrypt&Decrypt (x509) tests using openssl
test_openssl_encryptdecrypt_rsa_x509_rsa4096_512B
test_openssl_encryptdecrypt_rsa_x509_rsa4096_512B .................... PASSED

740 tests passed, 0 tests failed

Please let me know if this answers your question.

alcioa commented 2 years ago

Resolving the issue as answered.

aws / aws-nitro-enclaves-acm

How to successfully run tests via local docker development environment? #40

Building

Testing