search

notaryproject / notation

A CLI tool to sign and verify artifacts
https://notaryproject.dev/
Apache License 2.0
337 stars 84 forks source link

Improve the messages for `notation verify` #620

Open toddysm opened 1 year ago

toddysm commented 1 year ago

What is the areas you experience the issue in?

Notation CLI

What is not working as expected?

The following two output messages from notation verify look very similar although one is successful and the other is error:

$ notation verify $APPLICATION_IMAGE
Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:kubeconeu-demo-v1) because resolved digest may not point to the same signed artifact, as tags are mutable.
Successfully verified signature for ghcr.io/toddysm/flasksample@sha256:b4c0ed46d75ff7a1e8bcc166156db356dbb3768181d010a639ad4b4dd2f43db7
$ notation verify $TEST_IMAGE
Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:kubeconeu-demo-v1) because resolved digest may not point to the same signed artifact, as tags are mutable.
Error: signature verification failed: artifact "ghcr.io/toddysm/net-monitor@sha256:8c7a3bba4f87b1b9dc01e74b0a3b29cfa3fe492c320d536de87ce1ff67072442" has no applicable trust policy

What did you expect to happen?

We should rework those to make it obvious what is the result of the command. Right now they both start with the warning and this is the first thing the user sees. The user is more interested in the result of the command than the warning. An example can be:

SUCCESS: Successfully verified for signature for artifact ghcr.io/toddysm/flasksample@sha256:b4c0ed46d75ff7a1e8bcc166156db356dbb3768181d010a639ad4b4dd2f43db7

Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:kubeconeu-demo-v1) because resolved digest may not point to the same signed artifact, as tags are mutable.

and 

FAILURE: Failed to verify the signature for artifact "ghcr.io/toddysm/net-monitor@sha256:8c7a3bba4f87b1b9dc01e74b0a3b29cfa3fe492c320d536de87ce1ff67072442". There is no applicable trust policy for that scope.

Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:kubeconeu-demo-v1) because resolved digest may not point to the same signed artifact, as tags are mutable.

Adding white space and moving the warning to secondary provides better usability. Also, the current messages are not consistent in the language and not capitalized and grammatically correct.

How can we reproduce it?

Use the commands above

Describe your environment

N/A

What is the version of your Notation CLI or Notation Library?

Notation: Notary v2, A tool to sign, store, and verify artifacts.

Version: 1.0.0-rc.3 Go version: go1.20.1 Git commit: 233c0ea71edfb68b951eb54a739a101e2a05cd26

yizha1 commented 1 year ago

Move to post v1 release, as it is not critical for v1 release cc @vaninrao10

FeynmanZhou commented 4 months ago

Moved this issue to v1.2.0 milestone and linking #834 as we are improving all error messages in v1.2.0.

FeynmanZhou commented 4 months ago

Move the status message to the first line sounds reasonable because in general users care about the operation result at the first. Warning message is a follow-up reminder that users should be paying attention to.

For the operation status, I think it is not necessary to show SUCCESS or FAILURE at the beginning because when operation succeeded or failed, the error message starts with Successfully xxx or Error:. This is obvious for users.

How about improve the success and error messages as follows? 

Successfully verified for signature for artifact ghcr.io/toddysm/flasksample@sha256:b4c0ed46d75ff7a1e8bcc166156db356dbb3768181d010a639ad4b4dd2f43db7
Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:kubeconeu-demo-v1) because resolved digest may not point to the same signed artifact, as tags are mutable. 

Error: Failed to verify the signature for artifact "ghcr.io/toddysm/net-monitor@sha256:8c7a3bba4f87b1b9dc01e74b0a3b29cfa3fe492c320d536de87ce1ff67072442". There is no applicable trust policy for that scope.
Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:kubeconeu-demo-v1) because resolved digest may not point to the same signed artifact, as tags are mutable.
JeyJeyGao commented 4 months ago

Warning messages may be generated dynamically during the verification process. Any delay in displaying these warning messages will prevent users from knowing the current status of the verification. Are there any well-known CLIs that also delay showing warnings, so we can follow best practices? @FeynmanZhou @toddysm