`nitro-cli build-enclave` failing with DockerError

[gkelly]

If I use the nitro-cli that's packaged with the amazon-linux-extras install aws-nitro-enclaves-cli I can run build-enclave without issue (nitro-cli version reports Nitro CLI 1.0.9). But building the current main branch on the same EC2 instance that can successfully build with the packaged version (using the same arguments) results in:

./target/debug/nitro-cli build-enclave --docker-uri hello-enclave:1.0 --output-file what.eif
Start building the Enclave Image...
Docker error: PullError
[ E50 ] Docker image pull error. Such error appears when trying to build an EIF file, but pulling the corresponding docker image fails. In this case, the error backtrace provides detailed informatino on the failure reason.

For more details, please visit https://docs.aws.amazon.com/enclaves/latest/user/cli-errors.html#E50

If you open a support ticket, please provide the error log found at "/var/log/nitro_enclaves/err2021-04-05T21:34:26.317152048+00:00.log"

And the /var/log/nitro_enclaves/err2021-04-05T21:34:26.317152048+00:00.log file contains:

 Action: Build Enclave
  Subactions:
    Failed to build enclave
    Failed to build EIF from docker
    Failed to pull docker image: DockerError
  Root error file: src/lib.rs
  Root error line: 145
  Build commit: 4d51695

[gkelly]

I don't think there should be a pull involved in this build-enclave, as the image is fully local.

[gkelly]

Adding to this, if I use a --docker-uri that can be pulled then the build succeeds:

[ec2-user@ip-10-1-1-100 aws-nitro-enclaves-cli]$ ./target/debug/nitro-cli build-enclave --docker-uri hello-world:latest --output-file what.eif
Start building the Enclave Image...
Enclave Image successfully created.
{
  "Measurements": {
    "HashAlgorithm": "Sha384 { ... }",
    "PCR0": "71e7680ae9c52032a98293899d93315310ff4787fc9a312027407fed94120b544c08fe440cc6371ad488109e41ab05db",
    "PCR1": "c35e620586e91ed40ca5ce360eedf77ba673719135951e293121cb3931220b00f87b5a15e94e25c01fecd08fc9139342",
    "PCR2": "aa64ebcf9f8ec26838bade833b6dc47f145123b1ac56ba6b8990104ebf3d6b1a133e30c31a1560c37f1763502696320d"
  }
}

[gkelly]

Looking at the code this behavior looks fully intentional. From a user standpoint being able to reference a local image (and being the one responsible for getting your local image into the right state before the tool runs) seems preferable. Maybe I'm missing something here though.

[petreeftime]

build-enclave should work with local images as well. It's being used as such during testing as well. Will take a look.

[petreeftime]

I can verify that on the latest release it seems to not work.

[petreeftime]

It worked before PR https://github.com/aws/aws-nitro-enclaves-cli/pull/236, doesn't work after this PR.

[andraprs]

I'll take a look and come back with updates with regard to why it's this behavior.

[andraprs]

It seems that there is a change in the pull image flow after the shiplift crate update to v0.7.

Before shiplift crate update:

$ git diff
diff --git a/enclave_build/src/docker.rs b/enclave_build/src/docker.rs
index 41ba28b..5e40e0b 100644
--- a/enclave_build/src/docker.rs
+++ b/enclave_build/src/docker.rs
@@ -183,6 +183,7 @@ impl DockerUtil {
             // the most likely reason for failure when pulling, so log the
             // error.
             Err(err) => {
+                eprintln!("{:?}", err);
                 debug!("WARNING!! Credential could not be set {:?}", err);
             }
         };

$ ./build/nitro_cli/release/nitro-cli build-enclave --docker-uri hello:latest --output-file hello.eif
Start building the Enclave Image...
CredentialsError("No credentials found for the current image")
Enclave Image successfully created.
{
  "Measurements": {
    "HashAlgorithm": "Sha384 { ... }",
    "PCR0": "e033e86f364d1eaa35a84e66b3fdc91a8a55a1c53c3edabc5d82c74b4701fb08b24cdf6f85357adfa9b127ee608e56a7",
    "PCR1": "0e83514503485f818b4f4342774251376269082f311cd2debe4ab3d0b5f3c91a9957f1940f17d6bb07ed88434410bcfb",
    "PCR2": "cb359533d97535447027bff4779002974f440cb013443ca7909c107dbb1b2f7f9afa815ec95ec304b0ee3224457de6f4"
  }
}

After shiplift crate update:

$ git diff
diff --git a/enclave_build/src/docker.rs b/enclave_build/src/docker.rs
index 0000982..be9ab62 100644
--- a/enclave_build/src/docker.rs
+++ b/enclave_build/src/docker.rs
@@ -185,6 +185,7 @@ impl DockerUtil {
                 // the most likely reason for failure when pulling, so log the
                 // error.
                 Err(err) => {
+                    eprintln!("{:?}", err);
                     debug!("WARNING!! Credential could not be set {:?}", err);
                 }
             };
@@ -206,6 +207,7 @@ impl DockerUtil {
                         }
                         Err(e) => {
                             error!("{:?}", e);
+                            eprintln!("{:?}", e);
                             break Err(DockerError::PullError);
                         }
                     }

$ ./build/nitro_cli/release/nitro-cli build-enclave --docker-uri hello:latest --output-file hello.eif
Start building the Enclave Image...
CredentialsError("No credentials found for the current image")
Fault { code: 404, message: "pull access denied for hello, repository does not exist or may require \'docker login\': denied: requested access to the resource is denied" }
Docker error: PullError
[ E50 ] Docker image pull error. Such error appears when trying to build an EIF file, but pulling the corresponding docker image fails. In this case, the error backtrace provides detailed informatino on the failure reason.

For more details, please visit https://docs.aws.amazon.com/enclaves/latest/user/cli-errors.html#E50

If you open a support ticket, please provide the error log found at "/var/log/nitro_enclaves/err2021-04-06T09:14:00.422587488+00:00.log"

The additional messages come from these pieces of codebase, where I added error prints: https://github.com/aws/aws-nitro-enclaves-cli/blob/main/enclave_build/src/docker.rs#L187 https://github.com/aws/aws-nitro-enclaves-cli/blob/main/enclave_build/src/docker.rs#L207