search

oras-project / oras-go

ORAS Go library
https://oras.land
Apache License 2.0
184 stars 99 forks source link

Cannot Resolve descriptor: 404 not found while `oras blob fetch` does find it #823

Closed Skarlso closed 2 months ago

Skarlso commented 2 months ago

Basically what the title says.

I'm trying to resolve a layer like this but getting not found:

oras resolve ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller@sha256:2e3521c3642113452b802980672e7c4846c3f1069c5067c569d2bd64c7552764
Error response from registry: failed to resolve digest: ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller@sha256:2e3521c3642113452b802980672e7c4846c3f1069c5067c569d2bd64c7552764: not found

Yet when I run oras blob fetch, it's clearly there:

oras blob fetch --output - ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller@sha256:2e3521c3642113452b802980672e7c4846c3f1069c5067c569d2bd64c7552764
{"architecture":"amd64","config":{"User":"65534:65534","Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["notification-controller"],"Labels":{"org.opencontainers.image.created":"2023-08-23T14:47:13.967Z","org.opencontainers.image.description":"The GitOps Toolkit event forwarded and notification dispatcher","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"7de11b33d94c017dc2ad47e13f93fcf13667c4d4","org.opencontainers.image.source":"https://github.com/fluxcd/notification-controller","org.opencontainers.image.title":"notification-controller","org.opencontainers.image.url":"https://github.com/fluxcd/notification-controller","org.opencontainers.image.version":"v1.1.0"},"OnBuild":null},"created":"2023-08-23T14:56:17.420674442Z","history":[{"created":"2023-08-07T19:20:20.71894984Z","created_by":"/bin/sh -c #(nop) ADD file:32ff5e7a78b890996ee4681cc0a26185d3e9acdb4eb1e2aaccb2411f922fed6b in / "},{"created":"2023-08-07T19:20:20.894140623Z","created_by":"/bin/sh -c #(nop)  CMD [\"/bin/sh\"]","empty_layer":true},{"created":"2023-08-23T14:47:24.837095878Z","created_by":"ARG TARGETPLATFORM","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2023-08-23T14:47:24.837095878Z","created_by":"RUN |1 TARGETPLATFORM=linux/amd64 /bin/sh -c apk --no-cache add ca-certificates   \u0026\u0026 update-ca-certificates # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2023-08-23T14:56:17.420674442Z","created_by":"COPY /workspace/notification-controller /usr/local/bin/ # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2023-08-23T14:56:17.420674442Z","created_by":"USER 65534:65534","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2023-08-23T14:56:17.420674442Z","created_by":"ENTRYPOINT [\"notification-controller\"]","comment":"buildkit.dockerfile.v0","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:4693057ce2364720d39e57e85a5b8e0bd9ac3573716237736d6470ec5b7b7230","sha256:6bfe58a2bb4cf0e274f5a96a872acb132e45d159fcf4e887a2f5309eaf9dd650","sha256:a1668b593cbc6f971e61d4efd84455d77d6eccb02d33fcedf597c790900a47a4"]}}%

What am I doing wrong?

This is the manifest layer of that repo:

oras manifest fetch ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller:v1.3.0
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:49efb61aaec2b19664181995a22894102e206165f666c9280adfa7266a696ea4",
      "size": 865,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:3114fa44cf8aad5b1fdc44cd17aea7a9c09163d85621e833930593e534f2f926",
      "size": 865,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f1b641270009b4225363e72b7f5b1d90e1d4cadfa45f4d382cd68336ba73cab5",
      "size": 865,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f1a7bb3e6b44f4581bbf093262e3c566128d6cda5d0af78517a210e8ebf3d1a9",
      "size": 840,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:49efb61aaec2b19664181995a22894102e206165f666c9280adfa7266a696ea4",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:8a5780e99e75591c21ff6703621cdda59ded82cd89da478d16fdb19db6f0d5f0",
      "size": 840,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:3114fa44cf8aad5b1fdc44cd17aea7a9c09163d85621e833930593e534f2f926",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d11d59c3b22fbceba2df9fc9d47371e6ea36d856fb344cfe720de121561d967f",
      "size": 840,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:f1b641270009b4225363e72b7f5b1d90e1d4cadfa45f4d382cd68336ba73cab5",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}%
Skarlso commented 2 months ago

Doing it in code yields the same result:

    repo, err := remote.NewRepository("ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    desc, err := repo.Resolve(context.Background(), "sha256:2e3521c3642113452b802980672e7c4846c3f1069c5067c569d2bd64c7552764")
    if err != nil {
        fmt.Println("failed to resolve descriptor ", err)
                os.Exit(1)
    }

    fmt.Println(desc.Digest)

Fails too.

Skarlso commented 2 months ago

That's weird, I know just noticed that the SHA is in fact not under the listed layers. But then what the heck is it fetching when using this sha? :D :D :D What's going on here?

Skarlso commented 2 months ago

okay, I think I'm getting the wrong SHA somehow in my code. :/

This works:

func main() {
    repo, err := remote.NewRepository("ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    desc, err := repo.Resolve(context.Background(), "v1.3.0")
    if err != nil {
        fmt.Println("failed to resolve descriptor ", err)
    }

    fmt.Println(desc.Digest)
}

Looks like, this will get the right thing.

Wwwsylvia commented 2 months ago

Hi @Skarlso, oras resolve internally calls Repository.Resolve, which only resolves manifests.

Resolve resolves a reference to a manifest descriptor. See also ManifestMediaTypes.

To resolve a layer, you would need to explicitly specify the blob path by invoking repo.Blobs() like this:

    repo, err := remote.NewRepository("ghcr.io/open-component-model/mpas-bootstrap-component/fluxcd/notification-controller")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

-   desc, err := repo.Resolve(context.Background(), "sha256:2e3521c3642113452b802980672e7c4846c3f1069c5067c569d2bd64c7552764")
+       desc, err := repo.Blobs().Resolve(context.Background(), "sha256:2e3521c3642113452b802980672e7c4846c3f1069c5067c569d2bd64c7552764")

    if err != nil {
        fmt.Println("failed to resolve descriptor ", err)
                os.Exit(1)
    }

    fmt.Println(desc.Digest)
Skarlso commented 2 months ago

OMG THAT WORKED. :D Hum, I need to redesign things 🤔 Check out if I do fetch the right thing. But thanks, this really helped me go further. :) Thank you again :DDDD <3

Wwwsylvia commented 2 months ago

@Skarlso No problem. Good luck! 😀

Skarlso commented 2 months ago

I might need to read up on how OCI layers are structured better, but... could you help me out one last time, or point me to a doc where it makes a bit more sense so I structure my code better please? :)

The question is, when do I use repos Resolve vs Blob.Resolve? :D

Skarlso commented 2 months ago

Ah, so one lists blobs, and the other lists ManifestMediaTypes. So to list tags and such, I can use the repo Resolver to fetch things, but the actually get the blob data, I need to use Blobs()?

Wwwsylvia commented 2 months ago

@Skarlso The image-spec and the distribution-spec are what ORAS is following and are the source of truth. You may want to check manifest.md which describes the components that make up a container image.

The question is, when do I use repos Resolve vs Blob.Resolve? :D

Generally the API doc should help (just read it, LOL).

But the pattern is that:

BTW, welcome to join the #oras Slack channel to ask questions and get community news! 🙌

Wwwsylvia commented 2 months ago

Ah, so one lists blobs, and the other lists ManifestMediaTypes. So to list tags and such, I can use the repo Resolver to fetch things, but the actually get the blob data, I need to use Blobs()?

Not really, Repository.Fetch() can be used to fetch both manifests and blobs. You can take a look at the examples associated with it. 😀

But it does not harm to explicitly specify the path if you know what you want.

image

We would love to hear feedback on oras-go documentation! Please feel free to raise suggestions/complaints under #778 if you have any. 🙂

Skarlso commented 2 months ago

Ah thank you so much for the information!! That is a lot that can help here! :) Thank you! :) ❤️ ❤️