Proposal: buildx history command

tonistiigi commented 1 month ago

Description

Goals

Provide access to data stored by BuildKit via history record (provenance, logs, traces etc.) to docker buildx users. History APIs were added to BuildKit in version v0.11
Allow buildx users to better understand the internals of their builds
Allow replaying the logs without running the whole build again with --progress=plain (in default TTY mode logs disappear once a build step completes successfully). The same can be used to print logs without the multiplexing switches that happen when build steps run in parallel.

Build summary (could be separate PR)

Currently, when running buildx build or buildx bake in a machine where Docker Desktop is installed, “View build details: docker-desktop://dashboard/build/…” will be printed after build is complete allowing access to Docker Desktop Build Views. This should be replaced with more generic “summary” feature.

Summary should contain:

Duration of the build (especially important for --progress=plain that does not log this).
Link to open build in Docker Desktop (if it is installed)
Exported outputs. Eg. image names that were exported to docker or pushed to registry. Paths for local exports. Optionally in debug mode could even show example commands for running the image or inspecting with with buildx imagetools if it was pushed. If possible image can also be opened in Docker Desktop.
Example command to replay the logs. docker buildx history logs
Probably some summary of an error (eg. build step name that failed), but no need to duplicate whole error information again that is already printed.

Build completed: 21.0s (53 steps, 34 cached)
View build details: docker-desktop://dashboard/build/..
Outputs:
    Image    foobar                   (Exported to Docker)
    Image    docker.io/foo/bar:v2     (Pushed)
Replay logs: docker buildx history logs abc123def567

docker buildx history ls

Lists all the builds BuildKit history API knows about. Most recent builds are shown on top.

BUILD ID          NAME             STATUS          CREATED      DURATION           
abcd              binaries         Completed       5 min ago    1m03s        Open
abcef             binaries         Error           2 hours ago  3s           Open

Open link can be used to open build in Docker Desktop if supported.

Note that this command shows only the build for the active builder in Buildx, not like the build UI in Docker Desktop, that shows builds across all your builders.

docker buildx history logs [\]

Replay the logs for a specified build.

The logs would appear with the same formatting as --progress=plain , but there is no need to ever do multiplexing of build steps. Every step will only appear once in the output and print its whole log.

If <id> argument is not set, then the last build for the current builder will be shown. If there are no builds, then an error is shown.

docker buildx history inspect [\]

Print all details for a specified build. This includes data about duration, status, build request, outputs, build materials etc. With some --verbose flag, we could also print the whole graph of steps, but this can be left for a future iteration. Inspect output could end with examples of all other open/trace/logs etc commands for the same build.

Optionally, this command can be used to retrieve raw blobs from the build history, eg. the raw in-toto provenance, SBOM, image config, or opentelemetry trace. This can be done either with some use of --format flag or smth. like --type <mediatype>.

docker buildx history open [\]

Open specified build in Docker Desktop. This command requires Docker Desktop to be installed.

docker buildx history trace [\]

Open opentelemetry trace for the specified build.

In the initial implementation, this can run JaegerUI container (if one does not already exist), load the trace into it via POST request, and then open the link to the browser to view it. Optionally user could set their own URL to their JaegerUI instance with a flag. JaegerUI remains running, and users would need to call docker stop if they want to stop it.

docker buildx history rm [\...]

Delete a history record from the active BuildKit instance.

docker buildx history export [\…]

Export a specified build to a .dockerbuild archive that can be then imported to Docker Desktop (in another machine). This is the same archive currently generated by Docker’s Github Actions. The current implementation in Actions can start to just call buildx directly for this command.

docker buildx history import < file

Import a build into Docker Desktop from a .dockerbuild file created by history exportcommand or downloaded from Docker’s Github Actions. This command requires Docker Desktop to be installed as the store for imported builds is managed by Desktop(that is across builders). When completed, Buildx will show a link for opening the imported build in Desktop.

crazy-max commented 1 month ago

Should we have a buildx history prune command? I was looking at current buildx prune which clear caches. Was then wondering if buildx cache du/prune would make more sense.

tonistiigi commented 1 month ago

Should we have a buildx history prune command? I was looking at current buildx prune which clear caches. Was then wondering if buildx cache du/prune would make more sense.

buildx history rm could maybe be expanded with filters for multiple deletions. May be confusing to have multiple commands for deletion. Or maybe it makes sense to replace rm with prune.

docker / buildx