Open jedevc opened 2 years ago
I don't have anything against option 1, except the field should be Manifest
. This is the object Digest is referring to.
We can tackle the push part separately. I didn't really get Option3 as for anything more generic than imagetools create
push performs on a chain of object. If this isn't supported then it is more like a PushManifest
. I think I'd rather prefer setting manifest for the existing image exporter for pushing.
Sounds good :) I think option 1 will be good enough to get at least the inspect functionality to go through buildkit.
I think I made a typo above, we need to include the Manifest
in the response for sure. However, we also do need to make sure to provide a way to get the Index
, or at least a Platforms
field, so that we can enumerate all the platforms in the inspect.
Stumbled upon this ticket when I was searching for something related to docker manifest
. IIUC, this proposal was related to the discussion to move docker buildx imagetools
commands to the daemon-side?
I think we also need to have a good look where we want this to live; there's also docker manifest, which has a large overlap with imagetools. Question is; are all of these commands to be considered related to build or also for other purposes? Basically; should this be something to be provided by the engine API or the buildkit API (if we want it to be handled daemon-side?) Or would there be ways to expose (these parts of) the BuildKit API through the JSON remote API?
Hey folks, I think may have opened a ticket related to this, or at least further motivating the enhancements proposed here:
@thaJeztah , in your search for docker and manifests, perhaps you could look over my ticket and suggest any workarounds?
At the moment, to resolve images, the ResolveImageConfig API can be used - when passed an image ref and platform, the API returns the image digest of the manifest, and the contents of the config object.
@tonistiigi and @jedevc , I tried to developing a minimal go example to introspect what the ResolveImageConfig
API currently returns, but my experience in golang is a bit shallow. By contents of the config object
, would this preclude information such the list of layer digests?
This proposal suggests adding new/modifying existing APIs to extend the image resolution API.
Motivation
At the moment, to resolve images, the
ResolveImageConfig
API can be used - when passed an imageref
andplatform
, the API returns the image digest of the manifest, and the contents of the config object. However, this is not sufficient for all use cases: scenarios that involve listing all the available platforms by traversing the image index, or viewing layer information, or annotations attached at different levels than the config not currently possible.Additionally, because of this limitation, buildx must resolve images using methods imported from buildkit, instead of the API, resulting in the odd scenario where an image can be built+pushed using buildkit using registry information from
buildkit.toml
, but not inspected using the imagetools API.Option 1
Option 1 is the change of least resistance. We simply add new fields to
ResolveImageConfigResponse
that contains the index and manifest content:When querying an image config, we additionally attach the image manifest and index - this should add little overhead to the pull process, since we need to traverse to reach the image config anyways, however, depending on the size of the manifest and total number of manifests, each request may attach a lot of information that is never actually used.
Option 2
Option 2 requires some API modification and deprecation.
We rework the
ResolveImageConfigRequest
andResolveImageConfigResponse
to:This maintains a single endpoint with API compatibility with previous versions, but allows querying for not just the config, but the manifest or the image index. Additionally, this could easily be extended in the future with more precise queries for fetching annotations/attestations, etc.
While this maintains API compatibility, this requires replumbing large portions of existing go code, which is made harder by the necessity of returning the manifest digest, which only makes sense in the context of returning the config (since in the other cases, it is present either in the request, or the data response).
See a prototype here.
Option 3
Option 3 introduces a new set of APIs
Resolve
,Fetch
andPush
(for completeness), inspired by the containerd API.This is a generic fetch/push/resolve api, allowing querying and pushing from the client side. Additionally, this helps solve the additional buildx imagetools issue where manifests are created, and are pushed directly from the client, instead of through the server.
This places the traversal logic onto the client, and would allow implementing the
Resolver
API in the client, essentially allowing using buildkit as a pull-through tool.Conclusion
My personal preference is to implement option 3, and add the new APIs. This allows for far more flexibility than the other options in the future, and allows working from a clean design, that doesn't break any existing gRPC and Golang API contracts. Additionally, in the long term, the
ResolveImageConfig
API could be deprecated and reworked to be a shim layer over the new APIs.Comments appreciated, sorry for the long explanation, the options mostly walk through my thought process in thinking about solving this problem :tada: