Improve notation plugin error message

FeynmanZhou commented 8 months ago

What is not working as expected?

This issue summarizes the error messages of notation plugin that I suggest improving in v1.1.x.

Negative case: install a plugin on a non-matching CPU architecture machine

I installed Notation AKV plugin (Linux_arm64 version) on an Linux_amd64 machine with Notation v1.1.0 (main branch). The error response is empty in stderr as follows. This empty error response also exists when I installed the AWS Signer plugin in a mismatched Arch.

$ notation plugin install --file notation-azure-kv_1.0.1_linux_arm64.tar.gz
···
Error: plugin installation failed: failed to get metadata of new plugin: ERROR: response is not in JSON format. error: fork/exec /tmp/notation-plugin1835003109/notation-azure-kv: exec format error, stderr:

What did you expect to happen?

Suggested change:

By following the Notation error message guidance, notation should prompt users that the architecture of the installing plugin is not matched with the machine:

$ notation plugin install --file notation-azure-kv_1.0.1_linux_arm64.tar.gz
···
Error: plugin installation failed: failed to get metadata of new plugin. exec format error, stderr: non-matching CPU architecture
Use a matched architecture of the plugin and try to install it again

Or even give a much intelligent prompt with a solution:

$ notation plugin install --file notation-azure-kv_1.0.1_linux_arm64.tar.gz
···
Error: plugin installation failed: failed to get metadata of new plugin. exec format error, non-matching CPU architecture
Use a matched architecture linux/amd64 of the plugin and try to install it again

How can we reproduce it?

See above

Describe your environment

Linux, Golang 21.06

What is the version of your notation-go Library?

v1.1.0

JeyJeyGao commented 8 months ago

If you use a text file as the plugin executable, the OS will return the same error message. Therefore, we may not be able to determine the detailed error of fork/exec, whether it is an OS architecture issue or not.

How about error message for notation-go:

plugin installation failed: failed to execute the get-plugin-metadata command for azure-kv plugin: fork/exec /tmp/notation-plugin28956171/notation-azure-kv: exec format error

And add the suggestion in the CLI side:

Error: plugin installation failed: failed to execute the get-plugin-metadata command for azure-kv plugin: fork/exec /tmp/notation-plugin28956171/notation-azure-kv: exec format error. Please ensure the plugin is executable on the current OS and compliant with the Notation plugin protocol. If the error persists, try reaching out to the plugin maintainers for help.

FeynmanZhou commented 8 months ago

It seems not feasible to detect the OS/Arch information of the plugin during installation. Therefore, notation can only output generic error message regardless of in which error case under the current implementation. This is not helpful to notation CLI users.

Is it possible to enable notation detect the OS/Arch information of the plugin during installation in future releases? In this way, we can generate much more actionable suggestions in the error message to users.

FeynmanZhou commented 8 months ago

@JeyJeyGao

My suggested change for the error message from the notation CLI:

Error: plugin installation failed: failed to execute the get-plugin-metadata command for azure-kv plugin: fork/exec /tmp/notation-plugin28956171/notation-azure-kv: exec format error.
Please ensure the plugin is executable on the current OS and compliant with the plugin installation requirements. If the error persists, try reaching out to the plugin maintainers for help.

Note that there is no concept "Notation plugin protocol" existed in Notary Project or plugin vendors. In general, plugin vendors and Notary Project maintainers publish the plugin installation requirements on the website. So the most straightforward way here is to remind users to re-check and follow the plugin installation requirements accordingly.

JeyJeyGao commented 8 months ago

Error: plugin installation failed: failed to execute the get-plugin-metadata command for azure-kv plugin: fork/exec /tmp/notation-plugin28956171/notation-azure-kv: exec format error. Please ensure the plugin is executable and meet the installation requirements on the <OS/Arch>. If the error persists, try reaching out to the plugin maintainers for help.

How about this error message for Notation CLI? We can provide the OS and Arch in the recommandation.

FeynmanZhou commented 8 months ago

The prompted recommendation in notation CLI sounds good to me.

notaryproject / notation