oras pull should work against OCI layouts that have no tags but have no ambiguous manifests

[ziggythehamster]

What is the version of your ORAS CLI

1.1.0+Homebrew

What would you like to be added?

org.opencontainers.image.ref.name (i.e., tag) is not strictly required when working with an OCI layout directory, but most commands cannot work without either a tag or a sha256 reference (push is an example of one which does not require it).

My feature request is that oras pull (if not the other commands too) should just "do the right thing" if the OCI layout directory/tar has no tags and only has unambiguously one manifest that could be used (by that, I mean either it actually only has one manifest, or you've supplied --platform and filtered it down to exactly one manifest).

The alternative is to write this logic using jq and pass a sha256 reference, and that's unpleasant.

Why is this needed for ORAS?

ORAS should be able to read single-valid-manifest OCI layouts without tags, particularly given that it can create OCI layouts without tags.

Are you willing to submit PRs to contribute to this feature?

• [ ] Yes, I am willing to implement it.

[ziggythehamster]

I have figured out why this seems to be working this way and it might be a bug. If I have an OCI layout directory, even if I set a tag on all of my index.json manifest entries, ORAS only grabs the last element even if --platform would filter the list to exactly one. Using an @sha256:xxx suffix pinned to the manifest digest for the platform I expect works, but then --platform only checks the associated manifest, not the index.json, and thus the logic in SelectManifest expects the config to be a regular OCI image config. If I provide that correctly, then --platform works.

Going from the behavior of this, it's probable that the index-level platform filtering is expected to be done server-side, and the client-side cannot do it. So until this is corrected, the entire concept being proposed here makes zero sense. Also, --platform would not work without building a config file in the normal OCI config format.

It appears that this only applies to OCI directories; actual registries seem to work without issue (and those require a tag, which makes this feature request not relevant there).

[qweeah]

@ziggythehamster Regarding pulling without tag or digest, can you kindly share a real case, like what's in the blobs folder and what's the content of manifest.json, and how you expect oras pull should work?

Also --platform flag will 1) match a specific platform when the target manifest is an index or manifest list; 2) match the platform in the config if the target manifest is an image.

[qweeah]

Also, it would be great if you can share how you generate the OCI layout with only one manifest but no tag, we can look into it to improve our E2E experience in related tool eco-system.

[ziggythehamster]

can you kindly share a real case, like what's in the blobs folder and what's the content of manifest.json, and how you expect oras pull should work?

I'm currently writing tests for some GitLab CI automation that I'm working on, and when I filed this issue, I was trying to have an index.json that had three images:

• linux/arm64/v8:el8
• linux/arm64/v8:el9
• linux/amd64

It seems like all of the tooling struggles with this (e.g., nothing supports seeing os.version as a distinct platform, and instead gets mad there are multiple images in the layout). I have changed my tests to end up with a different OCI layout that only has two images - one aarch64 and one x86_64.

The real use case that I'm trying to support is that I want to have a Yum/DNF repo for EL8 and EL9 on both aarch64 and x86_64, but have all of these inside the same tag (because it's easier to have a private OCI registry that I can sync stuff down from than it is to have a private Yum/DNF repo over HTTPS). This is not possible since os.version isn't seen as a new platform by anything, so now I will have a tag for each EL version.

That said, as I was gathering some JSONs to put in here, I realize that I made a big mistake in how I'm combining OCI layouts - the index.json was referring to multiple manifests, but I actually need it to refer to a sub-index which refers to multiple manifests instead. I have a version of this working that is harder to maintain and what I'm working on now is to turn it into a GitLab component... and clearly I missed that :).

And then I would assume that --platform would work correctly. But I'll report back once I've fixed this.

share how you generate the OCI layout with only one manifest but no tag, we can look into it to improve our E2E experience in related tool eco-system.

Currently, there are no OCI tools that can do manifest operations against a local OCI layout directory... they all need a registry. umoci and manifest-tool almost do what I need (manifest-tool would work perfectly except it requires a registry). So I'm doing it with jq and some shell scripting.

Since there's no registry, there's no real need for tags, except that ORAS in particular seems to require them. Or maybe it doesn't and my wrong hierarchy is the problem. I'll let you know :).

[ziggythehamster]

Okay, so the above thing with --platform is a non-issue. Ignore all that noise. Back to the original issue.

This doesn't work:

$ oras pull --platform linux/arm64/v8:el8 --oci-layout foo --output bar
Error: foo: invalid image reference, expecting <name:tag|name@digest>

This selects exactly one image, though. If I add a tag with jq:

$ mv foo/index.json foo/orig.json
$ jq '.manifests |= map(.annotations["org.opencontainers.image.ref.name"] = "test")' < foo/orig.json > foo/index.json
$ oras pull --platform linux/arm64/v8:el8 --oci-layout foo:test --output bar
Downloading 98a17cf74ef8 foo.txt
Downloaded  98a17cf74ef8 foo.txt
Pulled [oci-layout] foo:test
Digest: sha256:04b0ecf41918379fadf6226b7708e8859bf2f27eb47538a921d328460526698c

If there were ambiguity, then obviously ORAS can't resolve the reference. It's probably sufficient to say that if index.json refers to exactly one thing, be it a manifest or an index, then the tag is not necessary in oras pull, since if you have multiple architectures, you will have an index referenced from index.json and still have one item (unless you're me and an idiot and forgot that you needed the extra level of indirection and wasted the poor maintainer's time with a non-issue :))

[qweeah]

What's the content of index.json before the piped jq command? Can you try below directly without running jq ...?

oras pull --platform linux/arm64/v8:el8 --oci-layout foo:test --output bar

[ziggythehamster]

$ oras pull --platform linux/arm64/v8:el8 --oci-layout oci-both-aarch64:test --output /tmp/orasout
Error: failed to resolve test: not found
$ skopeo copy --multi-arch all --dest-tls-verify=false oci:oci-both-aarch64 docker://localhost:5000/oci-both-aarch64:latest
Getting image list signatures
Copying 2 of 2 images in list
Copying image sha256:93865032310dc9e3ee27027553df7d4281bfdd22a202aa64a4130c0af13ce154 (1/2)
Getting image source signatures
Copying blob 98a17cf74ef8 skipped: already exists
Copying config a5be51b2dd done
Writing manifest to image destination
Copying image sha256:a15b21b7586bc37cd839eefe1bc4685a231007ea03c1e62742fe1fbbb9c11ef2 (2/2)
Getting image source signatures
Copying blob 92409e5fdcc9 skipped: already exists
Copying config 0e28853f6b done
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures

Here are the relevant files in the layout (I'll be censoring stuff so the sha256's won't match).

index.json:

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:2e14ac615a235914b1b25465549523e854c3e2d106e0a8647702856ed8394110",
      "size": 2319
    }
  ],
  "annotations": {
    "org.opencontainers.image.authors": "Keith Gable <xxx>",
    "org.opencontainers.image.created": "2023-12-15T01:51:58+00:00",
    "org.opencontainers.image.revision": "938955602e791ad7a8c6842de617062c921b90a1"
  }
}

blobs/sha256/2e14ac615a235914b1b25465549523e854c3e2d106e0a8647702856ed8394110:

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:93865032310dc9e3ee27027553df7d4281bfdd22a202aa64a4130c0af13ce154",
      "size": 1059,
      "annotations": {
        "author": "Keith Gable <xxx>",
        "org.opencontainers.image.authors": "Keith Gable <xxx>",
        "org.opencontainers.image.created": "2023-12-15T01:51:34+00:00",
        "org.opencontainers.image.revision": "938955602e791ad7a8c6842de617062c921b90a1"
      },
      "artifactType": "application/vnd.xxx.artifacts.v1",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8",
        "os.version": "el8"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:a15b21b7586bc37cd839eefe1bc4685a231007ea03c1e62742fe1fbbb9c11ef2",
      "size": 1059,
      "annotations": {
        "author": "Keith Gable <xxx>",
        "org.opencontainers.image.authors": "Keith Gable <xxx>",
        "org.opencontainers.image.created": "2023-12-15T01:51:36+00:00",
        "org.opencontainers.image.revision": "938955602e791ad7a8c6842de617062c921b90a1"
      },
      "artifactType": "application/vnd.xxx.artifacts.v1",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8",
        "os.version": "el9"
      }
    }
  ],
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "annotations": {
    "org.opencontainers.image.authors": "Keith Gable <xxx>",
    "org.opencontainers.image.created": "2023-12-15T01:51:58+00:00",
    "org.opencontainers.image.revision": "938955602e791ad7a8c6842de617062c921b90a1"
  }
}

The above was a test I just did to see if fixing the layout made --platform with an os.version work correctly, and it does work correctly with ORAS. Skopeo doesn't have an --override-os-version, so it just picks the first one, but it still copies it correctly. As you can see, as well, Skopeo accepts my no-tag-exists OCI layout directory but ORAS does not. I'm not a Go programmer, but I looked through the ORAS code, and it looks like ORAS would work without a tag too... but the CLI validation wants a tag.

[qweeah]

Thanks for the explanation. @ziggythehamster I think the best practice is to get the digest from the upstream tool that generates the multi-arch index and pull via:

oras pull --platform linux/arm64/v8:el8 --oci-layout foo@sha256:2e14ac615a235914b1b25465549523e854c3e2d106e0a8647702856ed8394110 --output bar

Why? The index.json should contain ALL manifests, which means an upstream image builder should also put linux/arm64/v8:el8 and linux/arm64/v8:el9 into the index.json.

So ideally, the index.json SHOULD contain 3 manifests(1 index + 2 specific-arch images), which produces ambiguity. The best practice is to pull via digest (or make upstream tool tag the multi-arch index if possible, buildx can)

[ziggythehamster]

In this case, I'm solely using ORAS, and what I'm doing is:

1. oras push to an OCI layout for each platform (in parallel, on different runners)
2. take each index.json, add the platform stuff, and merge the manifests in each into an index shoved into blobs
3. create a new index.json pointing at this index
4. eventually, this combined OCI layout is pushed to a registry, first to a tag like :mr-1234-1.2-gitsha, and once that MR is merged, to a tag like :1.2.

My goal with using platform tagging with ORAS is that I want the same OCI image to provide different sets of artifacts depending on the platform. This is very nearly how Homebrew uses OCI images, except that I'm wanting to use this for Yum/DNF repos in this case.

What I was previously doing - when --platform didn't work as I expected - was that I didn't shove the combined index into blobs and instead only had it as index.json. When doing that, every tool was seeing the OCI layout as having multiple images (and indeed, it had multiple images), so I would need to use jq to find the sha256 of the image I actually wanted based on filtering for platform. This doesn't push to a registry correctly (each thing would need its own tag - it wouldn't work like a multi-platform image does).

I think, however, that my use of ORAS is probably not how most people use ORAS - I'm not adding artifacts to existing images, but rather using OCI as a generic object store like Homebrew does. And functionally, I could just do what I need with skopeo and jq, but ORAS is more pleasant to use :).

[qweeah]

In this case, I'm solely using ORAS, and what I'm doing is:

1. oras push to an OCI layout for each platform (in parallel, on different runners)
2. take each index.json, add the platform stuff, and merge the manifests in each into an index shoved into blobs
3. create a new index.json pointing at this index
4. eventually, this combined OCI layout is pushed to a registry, first to a tag like :mr-1234-1.2-gitsha, and once that MR is merged, to a tag like :1.2.

We are working on improving this experience step-by-step. I think 2-4 fall into the composition scenario in issue https://github.com/oras-project/oras/issues/1053 and will be supported in the future. Also, to get the full reference of a pushed artifact, there is also a PR under reviewing: https://github.com/oras-project/oras/pull/1199

[qweeah]

oras push to an OCI layout for each platform (in parallel, on different runners)

BTW this is dangerous since index.json is not locked and some manifests might be overwritten unexpectedly.

[ziggythehamster]

I think 2-4 fall into the composition scenario in issue https://github.com/oras-project/oras/issues/1053 and will be supported in the future.

Ultimately, I would prefer to be able to create this natively with ORAS and #1053 would solve that, but I also recognize that what Homebrew and I are doing is different than attaching extra files to already extant images, and these would probably require different handling. You probably want oras image push (or similar) as a separate command, because there could be ambiguity around what exactly to do if you want to add multi-arch-aware artifacts to an existing container image.

Also, to get the full reference of a pushed artifact, there is also a PR under reviewing: https://github.com/oras-project/oras/pull/1199

This doesn't really work for my situation because if I want tag :1.2 to have many platforms inside it (vs. to have one tag per platform), I need an index.json that points at an index. It seems like the normal workflow in OCI is to push one tag per platform, and then use manifest-tool or similar to create a new index manifest at the "combined" tag that points at the other images, but this unnecessarily creates extra steps which can be avoided by just having one big OCI image with an index.json pointing at an index. It also means that you would have to be careful to avoid a race (by using @sha256:*) when tagging the "combined" image when you do push to a registry. manifest-tool ends up creating the same "index.json points at an index" structure as well.

BTW this is dangerous since index.json is not locked and some manifests might be overwritten unexpectedly.

They go to different subdirectories and are merged with a script in a subsequent CI job to avoid this:

# Combine OCI image layouts

export CURRENT_TIMESTAMP="$(date -u -Iseconds)"

mkdir -p ${OCI_OUT_DIR}/blobs/sha256
jq -nc '{ imageLayoutVersion: "1.0.0" }' > ${OCI_OUT_DIR}/oci-layout

# Check if either input's index.json points at an index already or not.
# If an index.json points at an index instead of a manifest, that will be what we use for this merge.

if jq -e '.manifests[0].mediaType == "application/vnd.oci.image.index.v1+json" or .manifests[0].mediaType == "application/vnd.docker.distribution.manifest.list.v2+json"' ${OCI_IN1_DIR}/index.json &> /dev/null; then
  export OCI_IN1_INDEX="${OCI_IN1_DIR}/blobs/$(jq -r '.manifests[0].digest' ${OCI_IN1_DIR}/index.json | tr ':' '/')"
else
  export OCI_IN1_INDEX="${OCI_IN1_DIR}/index.json"
fi

if jq -e '.manifests[0].mediaType == "application/vnd.oci.image.index.v1+json" or .manifests[0].mediaType == "application/vnd.docker.distribution.manifest.list.v2+json"' ${OCI_IN2_DIR}/index.json &> /dev/null; then
  export OCI_IN2_INDEX="${OCI_IN2_DIR}/blobs/$(jq -r '.manifests[0].digest' ${OCI_IN2_DIR}/index.json | tr ':' '/')"
else
  export OCI_IN2_INDEX="${OCI_IN2_DIR}/index.json"
fi

# Combine index files from the first and second input and add index-level
# annotations. Beware that one or both of these might have multiple entries
# already.
jq -s '
  .[0].manifests += .[1].manifests |
  .[0] |
  .mediaType = "application/vnd.oci.image.index.v1+json" |
  .annotations = {
    "example.gitlab.job.id": env.CI_JOB_ID,
    "example.gitlab.pipeline.id": env.CI_PIPELINE_ID,
    "example.gitlab.project.id": env.CI_PROJECT_ID,
    "org.opencontainers.image.authors": (env.GITLAB_USER_NAME + " <" + env.GITLAB_USER_EMAIL + ">"),
    "org.opencontainers.image.created": env.CURRENT_TIMESTAMP,
    "org.opencontainers.image.revision": env.CI_COMMIT_SHA,
    "org.opencontainers.image.version": env.PACKAGE_VERSION
  }
' ${OCI_IN1_INDEX} ${OCI_IN2_INDEX} > /tmp/index.json

# Delete the source indexes so we don't copy them if they're blobs
rm -v ${OCI_IN1_INDEX} ${OCI_IN2_INDEX}

# Copy blobs over from our input images
cp -v ${OCI_IN1_DIR}/blobs/sha256/* ${OCI_OUT_DIR}/blobs/sha256
cp -v ${OCI_IN2_DIR}/blobs/sha256/* ${OCI_OUT_DIR}/blobs/sha256

# Put the index into blobs
manifest_size=$(ls -l /tmp/index.json | awk '{ print $5 }')
manifest_sha256=$(sha256sum /tmp/index.json | cut -d ' ' -f 1)
mv -v /tmp/index.json ${OCI_OUT_DIR}/blobs/sha256/${manifest_sha256}

# Generate an index.json pointing at this index
jq -n --arg manifest_sha256 "${manifest_sha256}" --arg manifest_size "${manifest_size}" '{
  schemaVersion: 2,
  manifests: [
    {
      mediaType: "application/vnd.oci.image.index.v1+json",
      digest: ("sha256:" + $manifest_sha256),
      size: ($manifest_size | tonumber)
    }
  ],
  annotations: {
    "example.gitlab.job.id": env.CI_JOB_ID,
    "example.gitlab.pipeline.id": env.CI_PIPELINE_ID,
    "example.gitlab.project.id": env.CI_PROJECT_ID,
    "org.opencontainers.image.authors": (env.GITLAB_USER_NAME + " <" + env.GITLAB_USER_EMAIL + ">"),
    "org.opencontainers.image.created": env.CURRENT_TIMESTAMP,
    "org.opencontainers.image.revision": env.CI_COMMIT_SHA,
    "org.opencontainers.image.version": env.PACKAGE_VERSION
  }
}' > ${OCI_OUT_DIR}/index.json

[qweeah]

This doesn't really work for my situation because if I want tag :1.2 to have many platforms inside it (vs. to have one tag per platform), I need an index.json that points at an index.

You can push single-arch artifact without specifying a tag, like

oras push --oci-layout $OCI_IN1_DIR $file1 $file2 $file3

They go to different subdirectories and are merged with a script in a subsequent CI job to avoid this:

✅Well this is smart and should resolve the race condition.

The combination script can be simplified to one single oras command after https://github.com/oras-project/oras/issues/1053 completes, like:

oras index create $OCI_OUT_DIR:$TAG \
    # provide single-arch images
    --manifest=type=oci-layout,ref=$OCI_IN1_DIR@$OCI_IN1_DIGEST \
    --manifest=type=oci-layout,ref=$OCI_IN2_DIR@$OCI_IN2_DIGEST \
    $OCI_IN1_DIR@$OCI_IN1_DIGEST \
    # add annotations
    -a "example.gitlab.job.id"=env.CI_JOB_ID \ 
    -a "example.gitlab.pipeline.id"=env.CI_PIPELINE_ID \
    -a "example.gitlab.project.id"=env.CI_PROJECT_ID \
    -a "org.opencontainers.image.authors"=(env.GITLAB_USER_NAME + " <" + env.GITLAB_USER_EMAIL + ">") \
    -a "org.opencontainers.image.created"=env.CURRENT_TIMESTAMP \
    -a "org.opencontainers.image.revision"=env.CI_COMMIT_SHA \
    -a "org.opencontainers.image.version"=env.PACKAGE_VERSION

The missing piece is: how to get $OCI_IN1_DIR@$OCI_IN1_DIGEST and $OCI_IN2_DIR@$OCI_IN2_DIGEST from the tagless oras push run? IMHO you can get it via $OCI_IN1_DIR_AND_DIGEST=`oras push ... --format {{.Ref}}` as is suggested in #1199.

[qweeah]

@ziggythehamster Back to the request of this issue, it's more precise to pull with a digest reference and we are not going to support pulling without digest or tag. PRD #1199 describes how the digest can be easily obtained in next (v1.2.0) release.

If you are interested, let's follow up the discussion of index creation user experience in issue #1053.

[ziggythehamster]

For the above to work, #1066 would need to land, but otherwise I'm happy with that implementation, including needing to know the digest of the manifest that got "pushed" to an OCI layout directory.

we are not going to support pulling without digest or tag

In the case where you are pulling from an OCI layout directory whose index.json contains exactly one entry and that entry does not have an org.opencontainers.image.ref.name annotation, I don't see why this would need to be mandatory since there is unambiguously one valid image. The workaround would be this, which is inconvenient:

oras pull --platform linux/arm64/v8 --oci-layout foo@$(jq '.manifests[0].digest' < foo/index.json) --output bar

IMO, it's OK if this is non-default behavior behind an argument like --untagged or is a special tag like foo:@, foo:-, or foo: that would be invalid normally. And to reiterate, this is only applicable for OCI layout directories - OCI registries cannot have untagged images.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity.