Add manifest arch, os, and compressed layers size fields

git-hyagi commented 1 week ago

closes: #1767

ianballou commented 3 days ago

4. If we decided to create another django-admin command, we should align with Katello to see if it makes sense for them to run a management command reading data from the storage.

From the Katello side, we're okay with the command being the same as before or a new one. Choose whatever is most efficient from the Pulp side.

ianballou commented 3 days ago

Also I wanted to clarify to be super sure -- OCI Image Index manifests will always have a null arch, OS, and size? I'm not super clear since, according to https://specs.opencontainers.org/image-spec/image-index/?v=v1.0.1, the optional platform property can have an arch and an OS,

git-hyagi commented 2 days ago

Also I wanted to clarify to be super sure -- OCI Image Index manifests will always have a null arch, OS, and size? I'm not super clear since, according to https://specs.opencontainers.org/image-spec/image-index/?v=v1.0.1, the optional platform property can have an arch and an OS,

Thank you for bringing this! We reviewed and discussed the os and architecture fields from manifest list, and we will work on it in this PR. If the manifest list (or oci index) contains the platform (an optional) field, we will populate pulp manifest with arch and os (whenever platform is defined, os and arch are required).

git-hyagi commented 2 days ago

After re-reading the specs, I realized that manifestlist or oci-index do not have a platform field. Actually, platform is a field from manifests. In this link, we can see it better: https://github.com/opencontainers/image-spec/blob/main/image-index.md (platform is a bullet inside manifests).

So, please, ignore my last comment and yes, "OCI Image Index manifests will always have a null arch, OS, and size".

ianballou commented 2 days ago

After re-reading the specs, I realized that manifestlist or oci-index do not have a platform field. Actually, platform is a field from manifests. In this link, we can see it better: https://github.com/opencontainers/image-spec/blob/main/image-index.md (platform is a bullet inside manifests).

So, please, ignore my last comment and yes, "OCI Image Index manifests will always have a null arch, OS, and size".

Nice catch! I didn't notice it was under manifests, so that's perfect.

git-hyagi commented 1 day ago

If we need to store the "architecture" and "os" fields on the Manifest model, we should try to fetch the values from a manifest list first. An OCI Index has the "architecture" and "os" fields required. There is no need to read the data from a config blob if a manifest is listed within an index (https://github.com/opencontainers/image-spec/blob/main/image-index.md#image-index-property-descriptions).

I did some investigation in each workflow, and here are my findings:

for the sync and pull-through tasks, the first declarative content that we pass to the next pipeline stage is/are blobs/configblobs https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/tasks/sync_stages.py#L330-L345
- after that, in the resolve_flush method, we then send the manifests and manifest lists in this order: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/tasks/sync_stages.py#L274-L306
for the push workflow, we first push the blobs and the corresponding manifest for that image, and, just after pushing all images, the manifest-list/oci-index is pushed

I couldn't find how to fetch these values from the manifest list in advance. I'm not sure if I overlooked something or misunderstood the code workflow.

lubosmj commented 1 day ago

Mirroring: I believe you can extract the information about os/architecture inside the create_listed_manifest method: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/tasks/sync_stages.py#L480.
Pushing: When it comes to pushing, we may want to get this information from the already uploaded config blob: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/registry_api.py#L1295.

If we are fine (we are mostly not) with updating existing manifests, we could potentially update the objects here: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/registry_api.py#L1221.

git-hyagi commented 14 hours ago

Thank you for the help and the suggestions/optimizations!

pulp / pulp_container

Add manifest arch, os, and compressed layers size fields #1782