search

opencontainers / distribution-spec

OCI Distribution Specification
https://opencontainers.org
Apache License 2.0
823 stars 202 forks source link

Proposal: Extend Referrer API to image layer blob #517

Open njucjc opened 7 months ago

njucjc commented 7 months ago

According to the current definition of the Referrers API, it is limited to specifying one Manifest as the Referrers of another Manifest, but it does not apply to a specific OCI image layer. In some scenarios, it makes sense for one image layer to serve as an referrer pointing to another image layer.

For example, we may be able to use this feature in accelerated container image TurboOCI or SOCI scenarios to avoid repeated conversion of image layer index, thereby speeding up the generation time of accelerated container image layer indexes.

OCI Image                   Accelerated Image Index
  ｜                              ｜
  ｜--- Layer 1                   ｜--- Layer Index 1
  ｜                              ｜
  ｜--- Layer 2                   ｜--- Layer Index 2
  ｜                              ｜
  ｜--- Layer 3                   ｜--- Layer Index 3

When the image layer in the same registry is reused, the accelerated image layer index does not need to be re-produced. Specifically, our enhancement for the registry is as follows:

POST   /v2/<name>/blobs/uploads/?subject=<original image layer digest>
PUT    /v2/<name>/blobs/uploads/<reference>?subject=<original image layer digest>
PATCH  /v2/<name>/blobs/uploads/<reference>?digest=<accerated image index digest>?subject=<original image layer digest>

When uploading the blob for the accelerated container image layer index (e.g., Layer Index 1 digest) of the original image layer, include the digest of the original image layer it indexes (e.g., Layer 1 digest) in the additional subject field. Then, store the reference relationship between them in the image registry for use by the convertor.

When in use, obtain their reference relationship through the newly added blob Referrers API.

HEAD /v2/<name>/blobs/referrer?<original image layer digest>

With the aforementioned capabilities, when working on accelerating image index conversion, we can start by querying the blob referrers API of the image registry to determine if the remote repository contains the corresponding accelerated image layer index. If it does exist, we can skip the conversion for that layer. This is particularly valuable for large image layers, where the acceleration of index conversion can be time-consuming, so I think adding this capability is meaningful.

sudo-bmitch commented 7 months ago

There was discussion on this in https://github.com/opencontainers/image-spec/issues/1012

If the scenarios have layers already converted because of a shared base image, then it seems like reusing and extending the referrers of the base image may help in this scenario without any changes needed.