OCI images not supported

[everflux]

Describe the bug When having OCI images (f.e. created with Google Jib) in the registry, cleanup fails since the manifests could not be retrieved. Logoutput:

time="2021-09-11T11:07:34Z" level=error msg="Could not fetch tag!" repo=nbs/frontend tag=a259bf4
time="2021-09-11T11:07:34Z" level=error msg="Error obtaining tag data - skipping this repo" repo=nbs/frontend

To Reproduce Add OCI images to repository, run deckschrubber.

Expected behavior Cleanup of OCI images works.

Additional context To support OCI images at least the http accept header must include

application/vnd.oci.image.manifest.v1+json

The registry logs contain the following hint

127.0.0.1 - - [11/Sep/2021:11:07:34 +0000] "HEAD /v2/nbs/frontend/manifests/a259bf4 HTTP/1.1" 404 122 "" "Go-http-client/1.1"
time="2021-09-11T11:07:34.042602317Z" level=error msg="response completed with error" err.code="manifest unknown" err.message="OCI manifest found, but accept header does not support OCI manifests" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=380904d3-468c-4eec-a0d9-90592a37106f http.request.method=GET http.request.remoteaddr="127.0.0.1:47562" http.request.uri="/v2/nbs/frontend/manifests/a259bf4" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.158518ms http.response.status=404 http.response.written=122 vars.name="nbs/frontend" vars.reference=a259bf4 

The image is created with Jib in OCI image format, registry 2.7.1 is used.

[yan-foto]

This module relies on distribution to communicate with a registry. I have no experience with Google Jib or OCI images in general, but do they use the same API? Have you tried adding the header and see what happens?

[everflux]

Thanks for the quick reply! I checked the referenced module version, it says 2.7.1 - in the changelog of districution OCI support for 2.7.0 is explicitly mentioned: https://github.com/distribution/distribution/releases/tag/v2.7.0 This should be in 2.7.1 of distribution as well.

[yan-foto]

Do you care to submit a PR? I'm not going to be able to implement this any soon! Sorry!

[everflux]

Thanks for the update - I am not a go developer, unfortunately. Besides: I have no idea what the problem is, since you already reference the current distribution version which states that OCI is supported. So it might not be a simple dependency update after all.

[yan-foto]

If I understand the problem correctly (reading your logs), I suppose the answer is as easy as adding the custom headers to each HTTP request. This can be done using the context. For details, I also need to dive a bit deeper.

[elchenberg]

I changed the log output to get better error messages (e.g. tagLogger.Errorf("Could not fetch tag! %v", err)):

time="2022-06-13T10:45:01+02:00" level=error msg="Could not fetch tag! manifest unknown: OCI index found, but accept header does not support OCI indexes" repo=[...] tag=[...]

Then I had to add application/vnd.oci.image.index.v1+json to the Accept header to every request and it fixed the issue with fetching the tags of the OCI image.

But in the next step deckschrubber tries to fetch the manifests and this fails:

time="2022-06-13T10:55:23+02:00" level=error msg="Could not fetch manifest! unsupported manifest media type and no default available: application/vnd.oci.image.index.v1+json" repo=[...] tag=[...]

I tried to use application/vnd.oci.image.manifest.v1+json instead, as @everflux suggested, but fetching the tag failed:

time="2022-06-13T11:35:55+02:00" level=error msg="Could not fetch tag! manifest unknown: OCI index found, but accept header does not support OCI indexes"

Accepting both application/vnd.oci.image.index.v1+json and application/vnd.oci.image.manifest.v1+json did not help either.

I do not know how to continue and I am tempted to just skip tags with OCI images instead of skipping the whole repository. :see_no_evil:

Edit: Maybe I have a completely different problem. My OCI images are not real images but the buildkit cache and buildkit chose to use the media type application/vnd.oci.image.index.v1+json for those manifests instead of application/vnd.oci.image.manifest.v1+json. https://github.com/moby/buildkit/issues/2251

[everflux]

If you have a test-build I am happy to try it out!

[rwojsznis]

anyone managed to solve this? 🤔

ran into same headers-rabbit-hole with image build using buildx v0.11.2-desktop.4 and registry:2.8.2; as I'm running on 0 context here I tried randomly upgrading docker-related deps in the app itself (including distribution) to no avail 😅

[WhySoBad]

I ran into the same issue and as this project seems unmaintained I decided to create abwart. It's not a drop-in replacement for deckschrubber but it can be used for the same purposes by being deployed alongside a registry. Maybe this helps someone.