Proposal: introduce enhanced image resolution gateway API

[jedevc]

This proposal suggests adding new/modifying existing APIs to extend the image resolution API.

Motivation

At the moment, to resolve images, the ResolveImageConfig API can be used - when passed an image ref and platform, the API returns the image digest of the manifest, and the contents of the config object. However, this is not sufficient for all use cases: scenarios that involve listing all the available platforms by traversing the image index, or viewing layer information, or annotations attached at different levels than the config not currently possible.

Additionally, because of this limitation, buildx must resolve images using methods imported from buildkit, instead of the API, resulting in the odd scenario where an image can be built+pushed using buildkit using registry information from buildkit.toml, but not inspected using the imagetools API.

Option 1

Option 1 is the change of least resistance. We simply add new fields to ResolveImageConfigResponse that contains the index and manifest content:

message ResolveImageConfigResponse {
    string Digest = 1 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
    bytes Config = 2;
    bytes Index = 4;
}

When querying an image config, we additionally attach the image manifest and index - this should add little overhead to the pull process, since we need to traverse to reach the image config anyways, however, depending on the size of the manifest and total number of manifests, each request may attach a lot of information that is never actually used.

Option 2

Option 2 requires some API modification and deprecation.

We rework the ResolveImageConfigRequest and ResolveImageConfigResponse to:

message ResolveImageConfigRequest {
    string Ref = 1;
    pb.Platform PlatformDeprecated = 2;

    string ResolveMode = 3;
    string LogName = 4;
    int32 ResolverType = 5;
    string SessionID = 6;

    message ConfigType {
        pb.Platform Platform = 1;
    }
    message ManifestType {
        pb.Platform Platform = 1;
    }
    message IndexType {}

    oneof Type {
        ConfigType Config = 7;
        ManifestType Manifest = 8;
        IndexType Index = 9;
    }
}

message ResolveImageConfigResponse {
    string ManifestDigest = 1 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
    bytes Data = 2;
    string DataDigest = 3 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
}

This maintains a single endpoint with API compatibility with previous versions, but allows querying for not just the config, but the manifest or the image index. Additionally, this could easily be extended in the future with more precise queries for fetching annotations/attestations, etc.

While this maintains API compatibility, this requires replumbing large portions of existing go code, which is made harder by the necessity of returning the manifest digest, which only makes sense in the context of returning the config (since in the other cases, it is present either in the request, or the data response).

See a prototype here.

Option 3

Option 3 introduces a new set of APIs Resolve, Fetch and Push (for completeness), inspired by the containerd API.

message ResolveRequest {
    string Ref = 1;
    string Digest = 2 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
    bytes Data = 3;
}
message ResolveResponse {
    string Ref = 1;
    string Digest = 2 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
}

message FetchRequest {
    string Ref = 1;
    string Digest = 2 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
    ...
}
message FetchResponse {
    bytes Data = 1;
}

message PushRequest {
    string Ref = 1;
    string Digest = 2 [(gogoproto.customtype) = "github.com/opencontainers/go-digest.Digest", (gogoproto.nullable) = false];
    bytes Data = 3;
}
message PushResponse {}

This is a generic fetch/push/resolve api, allowing querying and pushing from the client side. Additionally, this helps solve the additional buildx imagetools issue where manifests are created, and are pushed directly from the client, instead of through the server.

This places the traversal logic onto the client, and would allow implementing the Resolver API in the client, essentially allowing using buildkit as a pull-through tool.

Conclusion

My personal preference is to implement option 3, and add the new APIs. This allows for far more flexibility than the other options in the future, and allows working from a clean design, that doesn't break any existing gRPC and Golang API contracts. Additionally, in the long term, the ResolveImageConfig API could be deprecated and reworked to be a shim layer over the new APIs.

Comments appreciated, sorry for the long explanation, the options mostly walk through my thought process in thinking about solving this problem :tada:

[tonistiigi]

I don't have anything against option 1, except the field should be Manifest. This is the object Digest is referring to.

We can tackle the push part separately. I didn't really get Option3 as for anything more generic than imagetools create push performs on a chain of object. If this isn't supported then it is more like a PushManifest. I think I'd rather prefer setting manifest for the existing image exporter for pushing.

[jedevc]

Sounds good :) I think option 1 will be good enough to get at least the inspect functionality to go through buildkit.

I think I made a typo above, we need to include the Manifest in the response for sure. However, we also do need to make sure to provide a way to get the Index, or at least a Platforms field, so that we can enumerate all the platforms in the inspect.

[thaJeztah]

Stumbled upon this ticket when I was searching for something related to docker manifest. IIUC, this proposal was related to the discussion to move docker buildx imagetools commands to the daemon-side?

I think we also need to have a good look where we want this to live; there's also docker manifest, which has a large overlap with imagetools. Question is; are all of these commands to be considered related to build or also for other purposes? Basically; should this be something to be provided by the engine API or the buildkit API (if we want it to be handled daemon-side?) Or would there be ways to expose (these parts of) the BuildKit API through the JSON remote API?

[ruffsl]

Hey folks, I think may have opened a ticket related to this, or at least further motivating the enhancements proposed here:

• https://github.com/moby/buildkit/issues/4854

@thaJeztah , in your search for docker and manifests, perhaps you could look over my ticket and suggest any workarounds?

At the moment, to resolve images, the ResolveImageConfig API can be used - when passed an image ref and platform, the API returns the image digest of the manifest, and the contents of the config object.

@tonistiigi and @jedevc , I tried to developing a minimal go example to introspect what the ResolveImageConfig API currently returns, but my experience in golang is a bit shallow. By contents of the config object, would this preclude information such the list of layer digests?

Pseudo Minimal Example (not working) ```go package main import ( "context" "fmt" "github.com/moby/buildkit/client" "github.com/moby/buildkit/client/llb" ) func main() { ctx := context.Background() // Create a new BuildKit client bkClient, err := client.New(ctx, "tcp://0.0.0.0:1234") if err != nil { panic(err) } // Define the build state := llb.Image("docker.io/library/alpine:latest") def, err := state.Marshal(ctx) if err != nil { panic(err) } // Solve the build _, err = bkClient.Solve(ctx, def, client.SolveOpt{}, nil) if err != nil { panic(err) } fmt.Println("Image built successfully") // Use the ResolveImageConfig API _, _, img, err := bkClient.ResolveImageConfig(ctx, "docker.io/library/alpine:latest") if err != nil { panic(err) } fmt.Printf("Image Config: %+v\n", img) } ```