opencontainers / distribution-spec

OCI Distribution Specification
https://opencontainers.org
Apache License 2.0
791 stars 199 forks source link

Allow registries to reject non-existent subjects in manifests #459

Open dmcgowan opened 11 months ago

dmcgowan commented 11 months ago

During conformance testing it was found that registries which require strong references between manifests and blobs fail conformance due to MUST language in the spec requiring acceptance of a manifest referencing a non-existent subject manifest. While subject fields may be described as a weak reference, listing and querying them at large scale may require a strong reference (such as foreign key in a database) or may simply be inheriting the data model used in 1.0 which always had referenced objects (as viewed from the merkle DAG) uploaded first.

The arguments for MUST language was to (1) support registries which may have reference only repositories, storing content elsewhere, and (2) ensure referrers exist at manifest pull time since there is no atomic way to upload referrers with manifests.

For (1) the burden will be on the client to handle this case on upload, as a registry is not required to support such repositories.

For (2) clients can retry or check for freshness when validation is a requirement or clients can ensure tags are only updated once all content is available. Similar issues have occurred in the past with multi-platform images. If images were uploaded before all platforms were available, then clients could see a race condition between the platform they need being built and the image they pull having that platform available. The same solution could apply here, use push by digest or a temporary tag when pushing manifests that should not be considered fully available and "tag" it once complete (via upload of the manifest using tag reference).

Changing the language the MUST to MAY makes most sense here. Additionally we can add guidance in the spec on how to perform manifest uploads more transactionally. In the future we could consider a more explicit way to create and manage transactions.

Related to https://github.com/opencontainers/distribution-spec/issues/340 https://github.com/opencontainers/distribution-spec/pull/341

mikebrow commented 8 months ago

There have now been 2 separate polls to resolve this issue:

  1. #482 (open to the public)
  2. #490 (limited to maintainers)

In both instances, the majority is in favor of not changing the current language in the specification.

This issue has been open now for 3 months. At this time I would like to request that @opencontainers/distribution-spec-maintainers respect the poll results and move forward with a release.

Yet the language in the specification is still changing, still has open PRs and both image and dist have known conflicts with regard to the MUST accept language being voted on.

Please respect the issue, review, and voting process (FYI votes such as 490,.. are not a simple majority they are 2/3). Let's see what Derek proposes.

As that work continues, we can "fix" the conflicts found many of which would be fixed the same way with or without MUST accept language, whatever that means.

jdolitsky commented 7 months ago

Any updates here?

sudo-bmitch commented 6 months ago

Based on the December 14th meeting, I believe we want to either close this or change the milestone so it's not blocking the 1.1 release. @dmcgowan are you in agreement with that and do you have a preference for which?

jdolitsky commented 6 months ago

Moved into the v1.2.0 milestone to clear out 1.1 as discussed on today's call