Closed git-hyagi closed 2 weeks ago
4. If we decided to create another django-admin command, we should align with Katello to see if it makes sense for them to run a management command reading data from the storage.
From the Katello side, we're okay with the command being the same as before or a new one. Choose whatever is most efficient from the Pulp side.
Also I wanted to clarify to be super sure -- OCI Image Index manifests will always have a null
arch, OS, and size? I'm not super clear since, according to https://specs.opencontainers.org/image-spec/image-index/?v=v1.0.1, the optional platform property can have an arch and an OS,
Also I wanted to clarify to be super sure -- OCI Image Index manifests will always have a
null
arch, OS, and size? I'm not super clear since, according to https://specs.opencontainers.org/image-spec/image-index/?v=v1.0.1, the optional platform property can have an arch and an OS,
Thank you for bringing this!
We reviewed and discussed the os
and architecture
fields from manifest list, and we will work on it in this PR.
If the manifest list (or oci index) contains the platform
(an optional) field, we will populate pulp manifest with arch
and os
(whenever platform
is defined, os
and arch
are required).
After re-reading the specs, I realized that manifestlist
or oci-index
do not have a platform
field. Actually, platform
is a field from manifests
. In this link, we can see it better: https://github.com/opencontainers/image-spec/blob/main/image-index.md (platform
is a bullet inside manifests
).
So, please, ignore my last comment and yes, "OCI Image Index manifests will always have a null arch, OS, and size".
After re-reading the specs, I realized that
manifestlist
oroci-index
do not have aplatform
field. Actually,platform
is a field frommanifests
. In this link, we can see it better: https://github.com/opencontainers/image-spec/blob/main/image-index.md (platform
is a bullet insidemanifests
).So, please, ignore my last comment and yes, "OCI Image Index manifests will always have a null arch, OS, and size".
Nice catch! I didn't notice it was under manifests, so that's perfect.
If we need to store the "architecture" and "os" fields on the Manifest model, we should try to fetch the values from a manifest list first. An OCI Index has the "architecture" and "os" fields required. There is no need to read the data from a config blob if a manifest is listed within an index (https://github.com/opencontainers/image-spec/blob/main/image-index.md#image-index-property-descriptions).
I did some investigation in each workflow, and here are my findings:
for the sync and pull-through tasks, the first declarative content
that we pass to the next pipeline stage is/are blobs
/configblobs
https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/tasks/sync_stages.py#L330-L345
resolve_flush
method, we then send the manifests
and manifest lists
in this order:
https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/tasks/sync_stages.py#L274-L306for the push workflow, we first push the blobs
and the corresponding manifest
for that image, and, just after pushing all images, the manifest-list
/oci-index
is pushed
I couldn't find how to fetch these values from the manifest list in advance. I'm not sure if I overlooked something or misunderstood the code workflow.
Mirroring:
I believe you can extract the information about os/architecture inside the create_listed_manifest
method: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/tasks/sync_stages.py#L480.
Pushing: When it comes to pushing, we may want to get this information from the already uploaded config blob: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/registry_api.py#L1295.
If we are fine (we are mostly not) with updating existing manifests, we could potentially update the objects here: https://github.com/pulp/pulp_container/blob/1a8fe634dba25b6adacd17459da3be931e37ae74/pulp_container/app/registry_api.py#L1221.
Thank you for the help and the suggestions/optimizations!
cc @sjha4 @qcjames53 we should keep an eye on this and integrate with it in Katello sooner rather than later to avoid excess reindexing of container manifests.
@git-hyagi, please, rebase this PR against the main branch and prepare it for the final review.
closes: #1767