Support OCI 1.1 GA specifications

yizha1 commented 4 months ago

Is your feature request related to a problem?

OCI v1.1 was released recently including:

The latest Notation release v1.1.0 uses image spec v1.1.0-rc.6 and by default use Referrers Tag schema when storing signatures. Besides uplifting dependency to the stable release v1.1 of the image spec, it is also recommended to revisit the experimental feature of "--allow-referrers-api flag" as registries will implement and enable Referrers API as OCI 1.1 is stable now. Some questions to answer:

Should we just remove --allow-referrers-api flag from experimental feature, and make it a stable feature?
Will Referrers Tag schema still be the default?
What are the timelines of popular registries supporting OCI v1.1? Compatibility with most registries should be our goal.

What solution do you propose?

N/A

What alternatives have you considered?

N/A

Any additional context?

N/A

No response

yizha1 commented 4 months ago

@priteshbandi @iamsamirzon We can discuss this issue for OCI 1.1 GA, and make a plan later.

ChristianCiach commented 4 months ago

Should we just remove --allow-referrers-api flag from experimental feature, and make it a stable feature?

Yes please :D

Will Referrers Tag schema still be the default?

Wouldn't this be a violation of the OCI spec? See https://github.com/opencontainers/distribution-spec/blob/main/spec.md

A client that pushes an image manifest with a defined subject field MUST verify the referrers API is available or fallback to updating the image index pushed to a tag described by the referrers tag schema. A client querying the referrers API and receiving a 404 Not Found MUST fallback to using an image index pushed to a tag described by the referrers tag schema.

Maybe I am reading this wrong, but it's my interpretation that the referrer-tag-schema is only valid as a fallback after a client tried to use the proper referrers-API first.

yizha1 commented 4 months ago

Should we just remove --allow-referrers-api flag from experimental feature, and make it a stable feature?

Yes please :D

Will Referrers Tag schema still be the default?

Wouldn't this be a violation of the OCI spec? See https://github.com/opencontainers/distribution-spec/blob/main/spec.md

A client that pushes an image manifest with a defined subject field MUST verify the referrers API is available or fallback to updating the image index pushed to a tag described by the referrers tag schema. A client querying the referrers API and receiving a 404 Not Found MUST fallback to using an image index pushed to a tag described by the referrers tag schema.

Maybe I am reading this wrong, but it's my interpretation that the referrer-tag-schema is only valid as a fallback after a client tried to use the proper referrers-API first.

It is the current default behavior of using referrer-tag-schema. The fallback process depends on registries that implement the referrers API per the OCI 1.1 spec, so that clients have a deterministic way to perform a fallback. However, this is not the case before OCI 1.1 spec release, which is also one of the reasons that referrer-tag-schema is the default.

sudo-bmitch commented 4 months ago

It is the current default behavior of using referrer-tag-schema. The fallback process depends on registries that implement the referrers API per the OCI 1.1 spec, so that clients have a deterministic way to perform a fallback. However, this is not the case before OCI 1.1 spec release, which is also one of the reasons that referrer-tag-schema is the default.

That wasn't my understanding of the subject/referrer specs. I was under the impression that clients would push an artifact with a subject, and if the registry did not indicate they supported the referrers API, the client would fall back to pushing the tag. The detection method has changed from a 404 on the referrers API to a header on the manifest push, but it hasn't depended on the release date of the 1.1.0 distribution-spec. Registries will transition on their own schedule, and potentially not at all. Hopefully I'm just misinterpreting.

cc @sajayantony

priteshbandi commented 4 months ago

Should we just remove --allow-referrers-api flag from experimental feature, and make it a stable feature?

Yes, now that OCI 1.1 is generally available (GA), we should remove this feature from the experimental flag. However, before doing so, can we please ensure that allow-referrers-api complies with the GA specification?

Will Referrers Tag schema still be the default?

Yes, unless we bump up major version.

What are the timelines of popular registries supporting OCI v1.1? Compatibility with most registries should be our goal.

IMO it shouldn't matter. The feature will be accessible through the allow-referrers-api flag and should align with the OCI 1.1 specification. Thus, if a registry supports OCI 1.1, it should function properly. Additionally, we can always make minor adjustments later if needed.

FeynmanZhou commented 4 months ago

I second with @priteshbandi 's opinions above.

My two cents:

Make sure the behavior of using --allow-referrers-api has the automatic fallback mechanism to referrers tag schema for backward consideration consideration.
Not all users are aware of the OCI Spec v1.1 changes and they might be unfamiliar with referrers api/referrers tag schema. If there are two stable options (referrers api and referrers tag schema) for users in Notation v1.2.0, we should at least document the scenarios and benefits of using each option.

yizha1 commented 4 months ago

Here is the summary per the discussion in the community meeting on Mar 4, 2024:

Current behavior

When experimental flag --allow-referrers-api is used, fallback to referrers Tag schema is enforced for all scenarios that referrers API does not work.

List of work items

[ ] Update Notary Project signature specification to comply with OCI 1.1 specs
[ ] Update --allow-referrers-api behavior to comply with OCI 1.1 distribution spec.
[ ] Review and decide update existing flag --allow-referrers-api to a stable flag
[ ] Review the default behavior of notation verify/ls/inspect as Referrers API supports returning preexisting image manifests that are listed in an image index tagged with the referrers tag schema. This will allow Notation to retrieve all the signatures no matter which method is used. Currently, if users do not specify --allow-referrers-api, only signatures using tag schema will be returned, and signatures stored with referrers API are missing.
[ ] Create a document for users to understand default behaviors and options

/cc: @priteshbandi @FeynmanZhou @shizhMSFT @vaninrao10 to review above work items

sudo-bmitch commented 3 months ago

Watching the recording of yesterday's meeting has me concerned. Does the lack of the --allow-referrers-api flag change the behavior of the tool to push an image without a subject field, and maintain your own tag, separate from the OCI referrers fallback tag? If not, it looks like the flag is pushing a decision on the user that was designed by the spec to be done automatically.

From the spec, when pushing a manifest, if the subject field exists, the tool doing the pushing must push the fallback tag when the registry does not include the header to indicate registry supports the API:

When pushing a manifest with the subject field and the OCI-Subject header was not set, the client MUST: ...

https://github.com/opencontainers/distribution-spec/blob/v1.1.0/spec.md#pushing-manifests-with-subject

When querying for referrers, the fallback is triggered on a 404 to the referrers API:

If the referrers API returns a 404, the client MUST fallback to pulling the referrers tag schema. The response SHOULD be an image index with the same content that would be expected from the referrers API. If the response to the referrers API is a 404, and the tag schema does not return a valid image index, the client SHOULD assume there are no referrers to the manifest.

https://github.com/opencontainers/distribution-spec/blob/v1.1.0/spec.md#listing-referrers

The upgrade policy for registries requires that they include content previously pushed to the fallback tag, so there's no need to merge both:

When registries add support for the referrers API, this API needs to account for manifests that were pushed before the API was available using the Referrers Tag Schema. ...

https://github.com/opencontainers/distribution-spec/blob/v1.1.0/spec.md#enabling-the-referrers-api

My concerns with making this a flag for the user (assuming that's switching whether the API or fallback tag is used) is that it creates a split brain issue where:

content pushed using the API will not be discovered by clients querying using the fallback tag
registries that support the API will still generate additional tags that clutter the tag namespace
users may encounter errors depending on the registry if they attempt to use the API when it is not available

yizha1 commented 3 months ago

Thanks @sudo-bmitch. I think your comments make sense if registries comply with OCI 1.1 spec. My understanding is if registries do not implement OCI 1.1 referrers API, clients do not have a deterministic way to do the fallback. This is the reason --allow-referrer-api was introduced and Referrers Tag schema was chosen as default behavior last year. With OCI 1.1 released and more registries implementing the referrers API, I think this flag --allow-referrer-api can be deprecated ultimately. /cc @shizhMSFT @priteshbandi for any comments.

sudo-bmitch commented 3 months ago

Hi @yizha1. The OCI-Subject header is used for the registry to indicate it supports the referrers API and no fallback is needed. This should not exist on a registry that does not implement the API, including the existing 1.0 registries.

They should also respond with a 404 or similar error to the referrers API request, which is the trigger to query the fallback tag. Registries that support the referrers API should always return a response, even if it is an empty list because the manifest doesn't exist or has no referrers.

We designed both of those to make it possible to automatically distinguish between a registry that supports the referrers API and an existing 1.0 registry, so that tooling would be able to gracefully fallback without requiring user interaction.

sudo-bmitch commented 3 months ago

To summarize my understanding from today's meeting:

The concern isn't that notation isn't able to know when a registry supports referrers, it's that older notation clients already exist that do not use the referrers API and only support the fallback tag. In other words, we are already in a split brain scenario where clients are depending on the fallback tag even when the registry is upgraded. So this flag is to ensure new versions of notation continue to support the older clients.

For a graceful upgrade, my suggestion is to switch the verification logic to always use the referrers API, falling back to the tag when a 404 or similar error is encountered. Then when clients later switch the push based on the registry support for referrers in the future, clients verifying will have hopefully already upgraded to support that. When pushing a manifest with a subject, there's no action to include it in the referrers API, so the registry is indifferent to seeing the fallback tag pushed.

priteshbandi commented 3 months ago

The high-level plan is as follows:

Update our current oci-1.1 implementation which is behind the export NOTATION_EXPERIMENTAL=1 flag to align with the GA version of the OCI Image and Distribution Specs v1.1.

In the notation 1.2 release:

Remove NOTATION_EXPERIMENTAL=1 environment variable need for --allow-referrers-api flag i.e. it won't be a experimental feature .
For the notation sign command, retain the --allow-referrers-api flag to allow users to choose between using the oci1.1 spec or the existing default tag schema-based implementation.
For the notation verify command, remove the --allow-referrers-api flag and implement automatic fallback using oci 1.1 spec, meaning check for the referrers API and if it returns a 404, look for the tag.

In future release, increase the major version for notation and make oci 1.1 as the default behavior for the sign operation.

PS:

The prerequisite for this plan is to identify the differences between our current referrers-based signing and verification implementation and the GA version of oci-1.1.
we probably need to rename --allow-referrers-api flag

cc: @sudo-bmitch @shizhMSFT @gokarnm @toddysm @yizha1

shizhMSFT commented 3 months ago

What's the default value of the --allow-referrers-api flag? Should it be default to --allow-referrers-api=true? That is, notation will use the Referrers API by default and fallback to Referrers Tag Schema if the remote registry does not Referrers API, which aligns to the OCI distribution v1.1.0.

yizha1 commented 3 months ago

Thanks @sudo-bmitch @priteshbandi @shizhMSFT for comments. According to the discussion in the meeting on Mar 12, 2024, Shiwei's comment was answered in the meeting. We are now aligned on the plan as Pritesh commented. Sajay posted a chat message during the meeting that for Notation v2.0.0, as referrers api is used by default, we may introduce a flag to provide an option for users to use referrers tag schema explicitly

sudo-bmitch commented 3 months ago

as referrers api is used by default, we may introduce a flag to provide an option for users to use referrers tag schema explicitly

From the discussion, the referrers API would always be used going forward, with an automatic fallback to the tag schema. The referrers API is queried on verification, not on signing. The flag would be to force the push of the fallback tag on signing, even if the registry indicates the referrers API is supported, to avoid breaking older clients that only know how to pull the tag.

yizha1 commented 3 months ago

as referrers api is used by default, we may introduce a flag to provide an option for users to use referrers tag schema explicitly

From the discussion, the referrers API would always be used going forward, with an automatic fallback to the tag schema. The referrers API is queried on verification, not on signing. The flag would be to force the push of the fallback tag on signing, even if the registry indicates the referrers API is supported, to avoid breaking older clients that only know how to pull the tag.

Agree. Thanks for the clarification.

notaryproject / notation