Explore OCI specification

dzikoysk commented 1 year ago

Request details

To support OCI repositories for e.g. Docker, we need to figure out how to do that. Specification:

https://github.com/opencontainers/distribution-spec/blob/main/spec.md

dzikoysk commented 1 year ago

If someone has some experience with it, it'd be nice to prepare list of routes with short description per each endpoint, so we could create a minimal working implementation on top of that.

TOTHT0MI commented 2 weeks ago

For future investigation I've collected resources, that can be useful. It seems pretty straight forward to implement, however the flow of uploading is quite different from Maven.

Official spec OCI Distribution spec: https://github.com/opencontainers/distribution-spec/blob/main/spec.md OCI Image spec: https://github.com/opencontainers/image-spec/blob/main/spec.md

Third party A summary of the specification: https://medium.com/@zbase000/oci-image-and-registry-specifications-a59db599991f

dzikoysk commented 2 weeks ago

Thanks, Docker support is something I'd really like to see in 4.x (at some point) :) I'll probably try to bootstrap some sort of proof of concept, or browse GitHub to check existing projects, to get a brief overview on how the minimal working impl would look like :pray:

zrdzn commented 2 days ago

I did some research and created descriptions for the endpoints, which should help visualize how we should approach the implementation of this specification, at least to some extent.

Definitions

<name> - refers to the namespace of the repository
<blob> - binary form of stored content
<digest> - unique identifier of a blob
<reference> - either a manifest's digest OR a tag (custom readable pointer to a manifest)
<location> - upload URL or session ID that was generated in 1st part of 2-step method
<manifest> - a configuration and set of layers for a single container image for a specific architecture and operating system. Read more here.

Blob Upload Strategies

We have the option to support two blob upload strategies. We can either support one or both at the same time, but this will certainly affect the implementation time.

1-STEP

The simplest option is to support a single request via POST, which uploads the blob to the registry. This is done by making a POST /v2/<name>/blobs/uploads/?digest=<digest> request, where we include the byte stream in the body and receive a location for the newly created blob in response.

Advantages

Easy and quick implementation
A good choice for small blobs

Disadvantages

No option to split the blob into parts, which can be problematic for large blobs
No option to resume the upload, which is disadvantageous for poor connections

2-STEP

This approach is slightly more complex. First, we make a POST /v2/<name>/blobs/uploads/ request, which returns a path or session ID that we use in the next PUT /v2/<name>/blobs/uploads/<location>?digest=<digest> request. The second request (PUT) is very similar to the request in the 1-STEP strategy.

Advantages

Allows splitting the blob into chunks
Allows resuming an existing upload
Suitable for uploads with poor connections
Suitable for large blobs

Disadvantages

Longer implementation time

Conclusions

I think we should implement both approaches due to the different needs they address. The specification mentions that some implementations do not support the 1-STEP option, so we could treat it as an optional feature. If we were to support only one option, it would likely be better to use the 2-STEP approach – it has greater long-term benefits.

Endpoints (Incompleted)

GET `/v2/`

Checks if registry implements specification.

Codes:

200
404
401

GET/HEAD `/v2/<name>/blobs/<digest>`

Finds a blob by providing a digest.

Codes:

200
404

DELETE `/v2/<name>/blobs/<digest>`

Deletes a blob by providing its digest.

Codes:

202
404
400
405

POST `/v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>`

Mounting a blob from another repository in the same registry. It will mount blob with provided digest to <name> repository, from repository <other_name>.

Response:

Headers
- Location - location of a newly mounted blob

Codes:

201
202
404

GET/HEAD `/v2/<name>/manifests/<reference>`

Finds a manifest by its digest or a tag.

Codes:

200
404

POST `/v2/<name>/blobs/uploads/` 2-STEP METHOD (1/2)

It is used in 2-step method which basically initiates an upload session and generates an upload URL or session ID, which is used for the subsequent upload or chunks.

Response:

Headers
- Location - upload URL/session ID
- (Optional) OCI-Chunk-Min-Length - minimum size of a chunk in bytes

Codes:

202

PUT `/v2/<name>/blobs/uploads/<location>?digest=<digest>` 2-STEP METHOD (2/2)

Uploads the blob (file or data) to the registry using the upload URL obtained in the 1st part of 2-step method.

Request:

Headers
- Content-Length - size of blob in bytes
- Content-Type - application/octet-stream
Body
- Byte stream

Response:

Headers
- Location - location of a newly created blob

Codes:

201
404
400

POST `/v2/<name>/blobs/uploads/?digest=<digest>` 1-STEP METHOD

Simply push blobs by a single request.

Request:

Headers
- Content-Length - size of blob in bytes
- Content-Type - application/octet-stream
Body
- Byte stream

Response:

Headers
- Location - location of a newly created blob

Codes:

201
202
404
400

PATCH `/v2/<name>/blobs/uploads/<location>`

Allows to upload blob by chunks instead of the single big one. They must however, be uploaded in correct order - it means that the <min> in chunk must be the last chunk's <max> + 1. Location must be generated from 1st part of 2-step method above.

Request:

Headers
- Content-Type - application/octet-stream
- Content-Range - inclusive range of bytes in format <min>-<max> (First chunk must start from 0)
- Content-Length - size of chunk in bytes
Body
- Byte stream of a chunk

Codes:

202
404
416 - chunks are uploaded out of order (ranges do not not connect to each other)

PUT `/v2/<name>/manifests/<reference>`

Uploads a manifest with a specified digest/tag.

Request:

Headers
- Content-Type - type of manifest being pushed e.g. application/vnd.oci.image.manifest.v1+json (Registry MUST ignore parameters key=value)

Body

mediaType in body MUST match the media type provided in Content-Type

{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
<rest of the content>
}

Response:

Headers
- Location - location of a newly created manifest

Codes:

201
404
413 - maximum size of manifest achieved

DELETE `/v2/<name>/manifests/<reference>`

Codes:

202
404
400
405

GET `/v2/<name>/tags/list`

Returns a list of existing tags for a given namespace. Response can contain empty list of tags. Tags MUST be in case-insensitive alphanumeric order.

Response:

Body

{
"name": "<name>",
"tags": [
"<tag1>",
"<tag2>",
"<tag3>"
]
}

Codes:

200
404

GET `/v2/<name>/tags/list?n=<amount>&last=<tagname>`

Returns a subset of tags with amount equal or less to provided in <amount>. When given <amount> is 0, response MUST contain empty list of tags. Additionally there is <tagname> parameter which defines from which non-exclusive tag should the tags be returned for <amount> tags above - provided tag in <tagname> will not be included in the response. When the <tagname> is provided, then the <amount> is optional.

Response:

Body

{
"name": "<name>",
"tags": [
"<tag1>",
"<tag2>",
"<tag3>"
]
}

Codes:

200
404

GET `/v2/<name>/referrers/<digest>?artifactType=<artifactType>`

Fetches a list of referrers by a manifest's digest. Descriptors MUST include artifactType field where the value matches to the manifest's artifact type - if there is none, just set artifactType to the mediaType value. Annotations MUST be taken from manifest. If there are no referrers, manifests list MUST be empty. Optionally client may want to filter referrers by artifactType, if the filter is present - the header OCI-Filters-Applied: artifactType is required.

Response:

Headers
- Content-Type - application/vnd.oci.image.index.v1+json
- (When artifactType query present) OCI-Filters-Applied - artifactType

Body

{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "size": <size of manifest>,
  "digest": <digest of manifest>,
  "artifactType": <artifact type>,
  "annotations": {
    "org.opencontainers.image.created": "2022-01-01T14:42:55Z"
  }
}
]
}

Codes:

200
400

GET `/v2/<name>/blobs/uploads/<location>`

Used when there was an upload of chunks, but they were out of correct order. It is required to get status of an active upload to resume it.

Response:

Headers
- Location - location provided in the url
- Range - 0-<position of last uploaded byte>

Codes:

204
404

dzikoysk / reposilite