Improve error messages for notation CLI

[nelson-wu]

Some examples:

$ notation cert generate-test
2021/11/12 13:53:30 missing certificate hosts

Perhaps flesh this out more with examples, similar to az cli

> az acr import -n 
the following arguments are required: --source

TRY THIS:
az acr import --name MyRegistry --source docker.io/library/hello-world:latest --image targetrepository:targettag
Import an image from a public repository on Docker Hub. The image uses the specified repository and tag names.

az acr import --name MyRegistry --source sourcerepository:sourcetag --image targetrepository:targettag --registry /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/sourceResourceGroup/providers/Microsoft.ContainerRegistry/registries/sourceRegistry
Import an image from an Azure container registry in a different subscription.

https://docs.microsoft.com/en-US/cli/azure/acr#az_acr_import
Read more about the command in reference docs

[dtzar]

We should get better error messages as a result of moving to cobra, but should evaluate the experience of error messages after the PR merges https://github.com/notaryproject/notation/pull/255

[dtzar]

Let's also in this improvement make sure the CLIs exit with proper exit code (non zero for failures) https://github.com/urfave/cli/blob/master/docs/v2/manual.md#exit-code

[iamsamirzon]

@dtzar - I created a new issue related to helping users deal with CLI errors. Refer https://github.com/notaryproject/notation/issues/300

[priteshbandi]

Some examples of inappropriate errors:

Missing trustpolicy.json file

➜  notation git:(main) ✗ ./notation verify $IMAGE
Resolved artifact tag `v1` to digest `sha256:36ca4d6834ed680362327811238b97c687e77c5cf4a04a74d0853d3c0c17e059` before verification.
Warning: The resolved digest may not point to the same signed artifact, since tags are mutable.
Error: open /Users/pritesb/Library/Application Support/notation/trustpolicy.json: no such file or directory

Empty trustpolicy.json file

➜  notation git:(main) ✗ ./notation verify $IMAGE
Resolved artifact tag `v1` to digest `sha256:36ca4d6834ed680362327811238b97c687e77c5cf4a04a74d0853d3c0c17e059` before verification.
Warning: The resolved digest may not point to the same signed artifact, since tags are mutable.
Error: EOF

trustpolicy.json file content: {}

➜  notation git:(main) ✗ ./notation verify $IMAGE
Resolved artifact tag `v1` to digest `sha256:36ca4d6834ed680362327811238b97c687e77c5cf4a04a74d0853d3c0c17e059` before verification.
Warning: The resolved digest may not point to the same signed artifact, since tags are mutable.
Error: trust policy document uses unsupported version ""

trustpolicy.json file content: hola

➜  notation git:(main) ✗ ./notation verify $IMAGE
Resolved artifact tag `v1` to digest `sha256:36ca4d6834ed680362327811238b97c687e77c5cf4a04a74d0853d3c0c17e059` before verification.
Warning: The resolved digest may not point to the same signed artifact, since tags are mutable.
Error: invalid character 'h' looking for beginning of value

[patrickzheng200]

Error: open /Users/pritesb/Library/Application Support/notation/trustpolicy.json: no such file or directory

@priteshbandi Is there any concern/suggestion regarding this error message? Since user needs to manually set up the trustpolicy.json file in current notation version, I think we need to explicitly print out the path when the file is missing.

[patrickzheng200]

Closing this issue as it's resolved.

[priteshbandi]

Reopening as are still inappropriate error messages

• [x] Notation not properly throwing the error from the plugin. The error is coming from this line. stderr needs to be converted to string from []byte before logging.

[yizha1]

Reopening as are still inappropriate error messages

• [x] Notation not properly throwing the error from the plugin. The error is coming from this line. stderr needs to be converted to string from []byte before logging.

@priteshbandi Is this issue solved by PR https://github.com/notaryproject/notation-go/pull/236?

[yizha1]

@priteshbandi could you confirm whether this issue was solved and released in rc.2? Thanks.

[priteshbandi]

Test result with notation rc2 build

➜  notation git:(main) ✗ ./notation cert generate-test
Error: missing certificate common_name
➜  notation git:(main) ✗ echo $?
1
➜  notation git:(main) ✗ ./notation cert generate-test --help
Generate a test RSA key and a corresponding self-signed certificate

Example - Generate a test RSA key and a corresponding self-signed certificate named "wabbit-networks.io":
  notation cert generate-test "wabbit-networks.io"

Example - Generate a test RSA key and a corresponding self-signed certificate, set RSA key as a default signing key:
  notation cert generate-test --default "wabbit-networks.io"

Usage:
  notation certificate generate-test [flags] <common_name>

Flags:
  -b, --bits int   RSA key bits (default 2048)
      --default    mark as default signing key

➜  notation git:(main) ✗ ./notation verify $IMAGE
Error: open /Users/pritesb/Library/Application Support/notation/trustpolicy.json: no such file or directory

➜  notation git:(main) ✗ ./notation verify $IMAGE
Error: malformed trustpolicy.json file

➜  notation git:(main) ✗ ./notation verify $IMAGE
Error: trust policy document is missing or has empty version, it must be specified

➜  notation git:(main) ✗ ./notation verify $IMAGE
Error: malformed trustpolicy.json file

[priteshbandi]

Apart from below error everything looks to be fixed

Error: open /Users/pritesb/Library/Application Support/notation/trustpolicy.json: no such file or directory

@priteshbandi Is there any concern/suggestion regarding this error message? Since user needs to manually set up the trustpolicy.json file in current notation version, I think we need to explicitly print out the path when the file is missing. The error message

IMO the error message Error: open /Users/pritesb/Library/Application Support/notation/trustpolicy.json: no such file or directory doesn't gives actionable user friendly message such as Trust policy is not present, please create trust policy at /Users/pritesb/Library/Application Support/notation/trustpolicy.json

[yizha1]

Linked to error message improvements work planned in v1.2.0, https://github.com/notaryproject/notation/issues/824.

[Two-Hearts]

Test result with Notation CLI v1.2.0-beta.1:

notation. verify $IMAGE
Error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy

[Two-Hearts]

Closing as this issue has been completed.