Open deitch opened 2 years ago
I can't understand how named contexts
or bake
can help. As mentioned in the Problem Description
buildkit with container-driver only supports loading FROM images via networked registries, i.e. FROM eve/foo:1.2.3 will only look in dockerhub from eve/foo:1.2.3. It does not look in the local cache, whether linuxkit's or docker's.
From my understanding whatever alias we have for eve/foo:1.2.3
linuxkit will still look in the docker hub for it.
I guess we can alias the name to something existing in the docker hub, but that is pointless, because the whole purpose of this efforts is to use something we build localy.
I am going to try and lay out the build issue with a practical example, and then how each potential solution impacts it.
Let's say we are a local developer or CI. We are making changes to lfedge/eve-uefi
, on which the downstream lfedge/eve-xen-tools
depends. So our Dockerfiles and build process look something like this:
# first build lfedge/eve-uefi:
$ linuxkit pkg build pkg/uefi
# if the current git tree hash for pkg/uefi is, e.g. 1234567abcd, then
# the above command will create an image - STORED LOCALLY - called "lfedge/eve-uefi:1234567abcd"
# next build lfedge/eve-xen-tools
$ linuxkit pkg build pkg/xen-tools
# if the current git tree hash for pkg/xen-tools is ffbb6677, then the above command will
# create an image - STORED LOCALLY - called "lfedge/eve-xen-tools:ffbb6677"
The file pkg/xen-tools/Dockerfile
has the line (generated by builds):
FROM lfedge/eve-uefi:1234567abcd
For the second build to work, the builder process must be able to find lfedge/eve-uefi:1234567abcd
, which is the output of the first.
We recall that lfedge/eve-uefi:1234567abcd
is the result of the first build. It never was pushed to a registry, and only exists locally.
The name lfedge/eve-uefi:1234567abcd
actually means, fully parsed, docker.io/lfedge/eve-uefi:1234567abcd
, i.e. that image on the (implicit) docker hub.
Because we are in a build process, actually testing changes, that image lfedge/eve-uefi:1234567abcd
cannot be pushed until we are done fully testing; it might never get pushed, if CI has issues and we changed the files to fix it.
Looking at the potential solutions.
That interim, might be temporary, image, lfedge/eve-uefi:1234567abcd
actually gets pushed to Docker Hub. This would ensure it always is available, but would pollute Docker Hub with lots of images, many of which were failures, risking someone actually using it.
In addition, this only would work for CI, which has credentials to push to docker.io/lfedge/*
. For developers working locally, who do not (and should not) have such credentials, it will not work.
In a "normal" docker build
, the line:
FROM lfedge/eve-uefi:1234567abcd
is interpreted as: "first look for lfedge/eve-uefi:1234567abcd
in my local image cache, and only go to the registry if I cannot find it."
This is useful behaviour for builds, and nearly every docker-based process depends upon it for the same reasons we do: build locally, FROM
locally, and only push when you are sure everything is good.
Unfortunately, this does not work with container-driver buildkit, which is the source of all of our problems. There is no way to tell container-driver buildkit, "go look in this cache before going to a registry".
In this case, even though the image name in the line is:
FROM lfedge/eve-uefi:1234567abcd
which means, "use the image lfedge/eve-uefi:1234567abcd
from Docker Hub", we use a local registry to store the image.
For this to work, the builder would need to know, "do not go to docker hub, but rather go to some local registry at localhost:8800
(or wherever)". The way to do that is to use build args in the Dockerfile:
ARG uefi=lfedge/eve-uefi:1234567abcd
FROM ${uefi}
A normal build of docker build
would just go to docker hub, but if you run docker build --build-arg uefi=localhost:8800/lfedge/eve-uefi:1234567abcd
, it will go get a different image.
The challenges are:
FROM
depends on another file that might change in development processThis solution should work, if somewhat messy.
I have wanted this for nearly a year. The same way docker has, "I first will look in my local cache, and only go registry if I cannot find it", I want buildkit to support the same thing. This is the best solution, both technically and operationally, but we are waging an uphill battle to make it happen.
Named contexts essentially let you alias an image on a FROM
line with a different source. So if the dockerfile is:
FROM lfedge/eve-uefi:1234567abcd
we could run docker build --build-context lfedge/eve-uefi:1234567abcd=<some other source>
. In this case, the builder would "override" getting lfedge/eve-uefi:1234567abcd
from docker hub, and instead get it from <some other source>
.
That some other source could be:
docker.io/lfedge/eve-uefi:1234567abcd
with quay.io/foo/other-image:tester
)For this to work, we would need:
These are just fancier versions of buildkit build contexts. Rather than specify it on the command-line with --build-context
, you put it in a bake file, which contains the aliasing.
For example, we would do:
target "xen-tools" {
contexts = {
"lfedge/eve-uefi:1234567abcd" = "<some other source>"
}
}
This then becomes the same question: where is <some other source>
and how do we get the image there?
bake-file is still required to publish to the registry or cache. This is a minimal example and commands: https://github.com/ruslan-zededa/lkt-try
You don't need a bakefile publish. buildx build
supports all sorts of output formats.
Someone had a good idea on using bake files, but only when dealing with local cache. I might try that, and if it works, get it into linuxkit.
It still becomes a headache. You would need to know at building the first image that you should "extra cache" it, and at the second image that you should source from that cache. What does that CLI look like?
lkt pkg build A --cache-local
lkt pkg build A --image-from A cache-local
Something like that?
This is ugly.
I may have a better solution. Still not perfect, but it looks better.
Recall the issue:
A
, lkt uses container-driver buildkit, and caches the result in its own OCI layout formatted cacheB
, lkt users container-driver buildkit, which looks for A
in registry, does not find it, errors outAs described above, buildkit has the ability to use build contexts (using build
or bake
) to "alias" images to other sources, including registries, local directories, git repos, etc.
If we add oci layout as a place to get it - which they have agreed with - then we could tell it, "go find A
in my local OCI layout cache".
The process then would look like:
lkt pkg build A # unchanged
lkt pkg build B --from-cache A # last flag added, meaning, "got get A from my linuxkit cache"
I am working on it. Once I fully understand the way to add a source, I will submit a PR to buildkit. It is a bit (a lot) challenging to grasp.
lkt pkg build A # unchanged lkt pkg build B --from-cache A # last flag added, meaning, "got get A from my linuxkit cache"
@deitch Can there be fallback so that we don't need to track whether A is in the build cache i.e., have it look in the cache first and if not found also look in registry?
I am +1 on @eriknordmark suggestion to the last @deitch idea -- @deitch what would it take to implement this in lkt?
Yeah, I think it can.
To recap:
Great @deitch ! Btw, what about cross builds in buildkit -- with what you're working on -- will we have that for free or is it still separate work?
Depends what you mean by cross builds. If you mean, "build for amd64 whole on arm64 using emulation", then, yeah, lkt already does it. It also can inherit remote builders for quicker builds. We just don't take advantage of it because of that issue. This will inherit all of that.
If you mean literal cross compiling, as in native go on amd64 compiling for arm64, it doesn't do that yet. I could get us there, but it is a lift.
Depends what you mean by cross builds. If you mean, "build for amd64 whole on arm64 using emulation", then, yeah, lkt already does it. It also can inherit remote builders for quicker builds. We just don't take advantage of it because of that issue. This will inherit all of that.
Yeah -- that's what I'm after -- can you please keep that usecase in mind while you're working on this issue? I can help test/etc. too -- really hope we can restructure our Dockerfiles/lkt/buildx setup in such a way that it would simply work out of the box. If not -- we may need to file additional issues I guess -- but that that kind of investigation is what I'm after.
The first buildkit PR is merged in!!
The first linuxkit PR is open and CI is green. More coming.
Linuxkit PR is merged in. Linuxkit now executes buildkit directly.
One more PR for linuxkit to support the cached build, and it is done.
This is a tracking issue for the chained builds problem. It is intended to be used as a place to track the issue and discuss possible solutions.
cc the following people who have been helping look into it: @eriknordmark @rvs @petr-zededa @ruslan-zededa . With many thanks to them.
Problem Description
Dockerfile
s. Each of these consumes other images inFROM
, sometimes multiple.FROM
images via networked registries, i.e.FROM eve/foo:1.2.3
will only look in dockerhub fromeve/foo:1.2.3
. It does not look in the local cache, whether linuxkit's or docker's.The above is the reason why the eve-os build process does not use the latest linuxkit. We cannot combine the docker-style "build an image, save it locally, build another that consumes that locally saved image" with the buildkit-style "build images across architectures". This is the catch-22
Scope
This only affects
Dockerfile
toDockerfile
"chained" builds. This does not affect final OS image build. The outputs of the variousdocker build
commands is consumable by linuxkit wherever it is.Impacted Files
In order to get a better handle on the scope of the problem, I ran a grep through all of lf-edge/eve to find what files are affected. Excluded vendor directories and our unique
.go
caching system.the results show 85
FROM
inDockerfiles
.Next, I excluded any that will always or never come from a registry, primarily upstream
alpine
,golang
,unikernel
,scratch
, simplifying it further:The results are down to 45 files.
Now we just need to see the value of
FROM
and how many times it is used:the results:
The
EVE_BUILDER_IMAGE
is set tolfedge/eve-alpine:6.7.0
in all cases, it just uses it as a build arg viaARG EVE_BUILDER_IMAGE=lfedge/eve-alpine:6.7.0
. So we can redo the results as:The problem arises when we want to build any one of those - primarily
lfedge/eve-alpine
- locally, and check its value by using it downstream. For example, if we make changes tolfedge/eve-alpine
, call itlfedge/eve-alpine:6.8.0-test
, and want to use it in other pkgs, the build would fail, as it is not pushed out to docker hub, nor should it be.In cases where we are just building the downstream packages that consume a well-known and published
lfedge/eve-alpine
, it will not be an issue, as it is downstream.The same is true for the interpolated value of
MKISO_TAG
andMKCONF_TAG
, etc. If we are building just the downstream packages, then the current value interpolated will already have been published. But if we are building the source ofMKISO
, and it has not been published yet, then the downstream build will fail.potential solutions
This section is for suggestions as to how to fix it.
Some starting points, none of which is necessarily good.
localhost:8800/lf-edge/eve-alpine:7.6.0
or similar in the dockerfile. This might work if we set all of them to use build args, but it is messy and requires a lot of extra overhead. It also is not clear that linuxkit supports build-args (although those could be added).Looking forward to insights.