Support BuildKit

[chris-crone]

Currently the Python SDK performs builds using the older build infrastructure. It would be good to support BuildKit so that users of the SDK can benefit from its various advantages.

[chris-crone]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the client side) and that service is distinctly non-trivial to implement.

[chris-crone]

Request for build secret support: https://github.com/docker/docker-py/issues/2174

[haizaar]

Request for build secret support: #2230

Did you mean to link to my original ticket? :)

[chris-crone]

Fixed, thanks @haizaar

[AkihiroSuda]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the client side) and that service is distinctly non-trivial to implement.

In addition to fssync, there are auth, content, secrets, and sshforward client-side services.

As @mipearson commented in https://github.com/moby/buildkit/issues/685 , docker-py should just invoke docker build CLI.

[tonistiigi]

My recommendation is to create a helper binary that compose can call as a quick practical solution. Note that fssync is optional for BuildKit, the Docker API can still receive tarballs fine and build them with BuildKit. It's less optimal but you still get the other benefits. But if you want secrets/ssh support then you do need either a helper or reimplement the grpc provider.

[AkihiroSuda]

Do we need a helper binary other than existing docker CLI?

[tonistiigi]

There are some things that might be interesting for compose that can be done in Go library and not in CLI atm. Like doing multiple requests in parallel with a shared session.

[mipearson]

Is there a possible future where most docker-compose things are done via a helper binary rather than via docker-py?

I seem to be having a very bad run of hitting docker-py specific issues lately :(

[tedivm]

Is this something people are working on? It sounds like a direction hasn't really even been picked yet.

[pmcanseco]

I have chosen to work around the lack of buildkit support by just calling the docker-cli in my projects. It works pretty well for me.

os.environ["DOCKER_BUILDKIT"] = 1
docker_build_command = 'docker build {} {}'
                       .format('--tag {} --target {} --rm=true --no-cache={} '
                               .format(tag, target, True),
                               path)
raw_build_result = subprocess.check_output(docker_build_command, 
                                           shell=True, stderr=subprocess.STDOUT,
                                           timeout=240)

[haizaar]

I have chosen to work around the lack of buildkit support by just calling the docker-cli in my projects.

Same here.

[tedivm]

I think that's what we're all doing, but it means that docker-compose is basically a nonentity at this point, and frankly docker-compose is much easier to use when setting up test environments. It's really unfortunate that such a useful tool is no longer usable due to this issue, and that there doesn't seem to be any timeline for resolving it (at this point I'm assuming docker compose will only get buildkit functionality when it becomes the default option).

[AkihiroSuda]

There is an exec-based PR for docker-compose: https://github.com/docker/compose/pull/6584

[adriantorrie]

There is an exec-based PR for docker-compose: docker/compose#6584

That PR has now been closed, and a new one opened: https://github.com/docker/compose/pull/6865

[abdennour]

Any deadline to release this feature and make it GA ? I am switching back to docker build which is very bad.

[FernandoMiguel]

for all the folks here have you seen https://github.com/docker/buildx ? I successfully replaced docker-compose build with it here's an example https://github.com/FernandoMiguel/BuildKit/blob/556808da7543cdb46f73853876c087628b6221dc/.github/workflows/build.yml#L137

[hrw]

Any update?

[AkihiroSuda]

https://github.com/docker/compose/pull/6865 got merged

I suggest closing this issue

[hrw]

How docker/compose is connected to docker/docker-py?

[AkihiroSuda]

docker/compose is the most widely known consumer of docker/docker-py. Other python projects should follow docker/compose (i.e. execute docker build command)

[hrw]

Then what the point of whole 'docker-py' if all projects have to switch to use Docker cli?

Your comment suggest "abandon all hope, forget about docker-py, use cli directly'.

[chris-crone]

Hi @hrw,

We'd gladly accept PRs for this and would help where possible. Note that this is tricky because of the issue mentioned above.

For Compose, the decision was taken to use the Docker CLI for build as it's unlikely someone will have Compose but not the CLI.

[hrw]

@chris-crone outside of my Python knowledge I suspect ;(

[FernandoMiguel]

not trying to be repetitive here, but this all already works out of the box with buildkit. just use buildx bake

[gabrieldemarmiesse]

Hi all, I made a Python client for Docker that sits on top of the Docker client binary (the one written in go). It took me several months of work. It notably has support for Docker buildx (build and bake at the moment). Actually the default call docker.build(...) uses buildkit/buildx underneath.

It's currently only available for my sponsors, but It'll be open source with an MIT licence May 1st, 2021 🙂

https://gabrieldemarmiesse.github.io/python-on-whales/

[gabrieldemarmiesse]

Hi all, in the end, making Python-on-whales pay-to-use wasn't a success. So I've open-sourced it.

It uses buildx (so with buildkit as backend) by default to build images.

It's free and on Pypi now. Have fun 😃

$ pip install python-on-whales
$ python
>>> from python_on_whales import docker

>>> my_image = docker.build("./")
[+] Building 1.4s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                             0.1s
 => => transferring dockerfile: 32B                                                                              0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                  0.0s
 => [internal] load metadata for docker.io/library/python:3.7                                                    0.0s
 => [make-sources 1/7] FROM docker.io/library/python:3.7                                                         0.0s
 => [internal] load build context                                                                                0.1s
 => => transferring context: 111.37kB                                                                            0.1s
 => CACHED [make-sources 2/7] COPY requirements.txt /                                                            0.0s
 => CACHED [make-sources 3/7] RUN pip install keras-autodoc -r /requirements.txt                                 0.0s
 => CACHED [make-sources 4/7] WORKDIR /python-on-whales                                                          0.0s
 => CACHED [make-sources 5/7] COPY . .                                                                           0.0s
 => CACHED [make-sources 6/7] RUN pip install -e .                                                               0.0s
 => CACHED [make-sources 7/7] RUN cd docs && python autogen.py                                                   0.0s
 => exporting to image                                                                                           1.2s
 => => exporting layers                                                                                          1.2s
 => => writing image sha256:967755340bad79c6b222b9470729a4d95513cab51ea881c8625980034f7c7678                     0.0s

>>> output = docker.run(my_image, ["echo", "Hello world!"])

>>> print(output)
Hello world!

There is support for nearly all buildx commands and flags.

docker.buildx.build(..., cache_from="my_registry/project:cache")

for example for remote layer caching. There are secrets, multi-platform images

docker.build(...., platforms=["linux/amd64", "linux/arm64"])

bake, making builders, ssh, pushing, etc...

Link to the documentation for docker.buildx.build(...) Link to the github repo

I hope you like It!

[aguckenber-chwy]

BTW our group needs this as well. Mainly we need the --secret support to docker buildx which new buildkits provide.

[shpml]

BuildKit is the default builder for users on Docker Desktop, and Docker Engine as of version 23.0. Is there any update on this feature request?

[ashpreetbedi]

Any idea if this is on the priority list?

[naufalafif]

this lib/sdk uses docker engine API & currently there's no specific argument for it. other way to do it is by making buildkit as default builder. refer https://docs.docker.com/build/buildkit/#getting-started

[lessismordaunt]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the client side) and that service is distinctly non-trivial to implement.

Is a pure-Python implementation strictly mandatory? Can we not depend on the Go implementation of this service while maintaining the spirit of Docker-Py? It seems low-level enough. What about using https://github.com/go-python/gopy for the binding layer?

BuildKit is pretty mandatory these days and I'd rather not revert to subprocess calls.

[beardedfoo]

Bump: I'm really shocked to see this in 2024:

  File "/usr/local/lib/python3.12/site-packages/docker/models/images.py", line 304, in build
    raise BuildError(chunk['error'], result_stream)
docker.errors.BuildError: the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled

TL;DR: It is not possible to build a modern Dockerfile using docker-py.

It's time to choose any option, not a convenient or nicely designed option, and fix this.

[acederberg]

Will this ever be fixed? The docker cli insists that I not use DOCKER_BUILDKIT=0 since builds without buildkit will be deprecated. This is shown with the following command

DOCKER_BUILDKIT=0 docker build .

which outputs

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

So when buildkit is removed this library be broken - Or am I mistaken?

[tkrah]

Is there an upstream reference about this deprecation somewhere with a plan / dates when this is going to happen (the official docs just says it's deprecated but no further dates are given when it will break)?

E.g. the docker-java API does also not support buildkit and a lot of projects use that one too .... That will break many users / consumers if non buildkit support will be removed.

[ilons]

Is there an upstream reference about this deprecation somewhere with a plan / dates when this is going to happen (the official docs just says it's deprecated but no further dates are given when it will break)?

As I understand it, the "legacy builder" was deprecated in docker cli 23.0 and have not yet gotten a set release when it will be removed. Given that 23.0 was released in february 2023 and are still supported, I would expect the 26.x release to be supported for at least another year.

Regardless, there is a sense of urgency to start the work adding buildkit support to this library if it wants to survive.

[coleary-hyperscience]

I have chosen to work around the lack of buildkit support by just calling the docker-cli in my projects. It works pretty well for me.

os.environ["DOCKER_BUILDKIT"] = 1
docker_build_command = 'docker build {} {}'
                       .format('--tag {} --target {} --rm=true --no-cache={} '
                               .format(tag, target, True),
                               path)
raw_build_result = subprocess.check_output(docker_build_command, 
                                           shell=True, stderr=subprocess.STDOUT,
                                           timeout=240)

Is the recommended immediate workaround to add docker build secrets still this?