search

microsoft / Windows-Containers

Welcome to our Windows Containers GitHub community! Ask questions, report bugs, and suggest features -- let's work together.
MIT License
396 stars 60 forks source link

As a Developer I want to be able to build Windows Containers with BuildKit #34

Open bplasmeijer opened 4 years ago

bplasmeijer commented 4 years ago

The windows BuildKit is only supported on Linux

image

jsturtevant commented 4 years ago

It is possible to use buildx (which uses buildkit) if there are no RUN commands in the docker file: 

docker buildx create --name img-builder --use
docker buildx build --no-cache --pull --push \
  --platform "windows/amd64" -t $imagetag -f $dockerfile .
docker buildx rm img-builder
ARG nanoserverTag="mcr.microsoft.com/windows/nanoserver:1809-amd64"
# othertags list: curl -L https://mcr.microsoft.com/v2/windows/nanoserver/tags/list

FROM ${nanoserverTag}
COPY some.exe "/Windows/System32/some.exe"

It would be interesting to see if could use some windows emulator connected to Binfmt_misc as QEMU is used for other architectures

jorgearteiro commented 4 years ago

There is an issue logged on BuildKit repo https://github.com/moby/buildkit/issues/616

TBBle commented 4 years ago

@jsturtevant If you wanted to go down that path, I believe Wine can be used with binfmt_misc, although I'm not sure how well that would work with namespaces since there's a WineServer and configuration directory to create, and you probably don't want that in the final image...

It would be easier to prototype that on a raw Linux box than via the Linux support in Docker on Windows, I think.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.

TBBle commented 3 years ago

Current status of my work on https://github.com/moby/buildkit/issues/616 is that it's stalled waiting on a pile of 4 containerd PRs to be reviewed and advanced. The top of the stack is https://github.com/containerd/containerd/pull/4419, but there's some outstanding design questions on the top two PRs regarding HCS behaviours, see https://github.com/microsoft/hcsshim/issues/853, but the bottom two are just pending containerd maintainer review to merge or rework as appropriate.

And after that, will come fixing path-related issues in the BuildKit LLB layer, for which I'd only scratched the surface of in https://github.com/moby/buildkit/pull/1621, exploring the solution space, and which needs to be reworked per maintainer feedback.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.

TBBle commented 3 years ago

Status unchanged, still waiting on merging or feedback on containerd support for Windows container creation/export/mount support, and then continuing the work on BuildKit itself, starting from https://github.com/moby/buildkit/pull/1621

immuzz commented 3 years ago

@kevpar Do you have any information on the containerd PRs mentioned above?

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.

TBBle commented 3 years ago

Still working through containerd PRs per above. One of four has landed, and one needs to be rewritten or at least iterated upon further. I've also opened PR on hcsshim again to remove some duplication of code with the containerd PRs.

That said, Docker is looking at supporting containerd on Windows for 21.x (https://github.com/moby/moby/issues/41455), which might mean we get BuildKit on Docker+containerd even without that containerd PR stack, as the parts I was trying to implement in containerd (WCOW layer management) are provided by Docker when Docker uses containerd, as far as I know.

I haven't looked closely at how BuildKit integrates with Docker though, it's possible it'll still need fixes in containerd, and either way, BuildKit itself has higher-level issues to be resolved, e.g., https://github.com/moby/buildkit/pull/1621.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.

TBBle commented 3 years ago

Not much has changed since December. The containerd PRs are now in a reasonable state, two are ready-to-go and one (https://github.com/containerd/containerd/pull/4419) is functional but possibly needs some discussion about the test suite changes, and depends on a hcsshim feature PR (https://github.com/microsoft/hcsshim/pull/901).

Some recent testing suggests that that hcsshim PR might have issues with Windows 10 20H2 that don't appear on Windows Server LTSC2019, but I haven't definitely proved that as I have only tested the latter via CI pipelines so far. Edit: Still using CI pipelines, I confirmed that the PR works on RS5 on AppVeyor (VS 2019 image), but fails on my personal Windows 10 20H2 box, suggesting that some part of the reverse engineering used to build that PR was incorrect or incomplete.

immuzz commented 3 years ago

@weijuans-msft can you provide clarity here as part of your effort to provide containerd parity with Docker developer experience.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

TBBle commented 3 years ago

Compared to January, I've identified and have a fix for the issue that https://github.com/microsoft/hcsshim/pull/901 shows on Windows 10 20H2, but that PR is still pending merge, and then the fix will be up for PR.

The containerd PRs (https://github.com/containerd/containerd/pull/4399, https://github.com/containerd/containerd/pull/4415, and https://github.com/containerd/containerd/pull/4419) should all be in a reasonable state to flow through, although the latter depends on https://github.com/microsoft/hcsshim/pull/901, often triggers a now-known containerd issue (https://github.com/containerd/containerd/issues/4924) in its test suite, and also frequently triggers an unknown (hcsshim?) issue that I've yet been unable to replicate outside containerd.

It's possible that the Docker/containerd integration for Windows (https://github.com/moby/moby/issues/41455 is the best place to track that) will mean some or all of those PRs stop being critical-path for BuildKit on Windows, and then the BuildKit-local platform issues (i.e. filesystem handling and similar) can be addressed.

weijuans-msft commented 3 years ago

@TBBle great work here! I changed this to "In Progress" as I am seeing you and lots of others in the moby and containerd community working on this. Thank you.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

grochoge commented 3 years ago

Without this, is there any good way to handle build-time secrets for Windows containers?

TBBle commented 3 years ago

One kind-of heavy-weight option, used by ue4-docker, is running a service outside the container that exposes the secret, and have the container image build pull that secret over the network connection.

You can see ue4-docker's implementation of the external host and a Git Credential Helper to query it, including a generated token to provide the bare-minimum protection.

A significant cost to this approach is that the token ends up in the image as a build-arg, so layer caching is confounded; I don't know if the same thing happens when using BuildKit secret mounts, but I assume not.

In ue4-docker's case, the container builds are done under control of the script, so the credential endpoint is served by the script, rather than being left running all the time, which make this approach more feasible than it would be for just a Dockerfile being distributed for use with docker build.

ghost commented 3 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

meirgbinahai commented 3 years ago

I'd love to assist, as I really need this feature to speed up the process of building Docker Windows images.

Is there any way to build stages in parallel in Windows without using BuildKit?

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

viceice commented 2 years ago

Can somebody please freeze this issue? 😕

weijuans-msft commented 2 years ago

I don't know if we can freeze issues. If anyone knows, let me know :).

I could use help to be enlightened on the benefit or the pain points this can address. I see these so far in this thread:

What else?

How does this help in your or your company's modernization effort? Is this really Developer centric or it can add value to your DevOps in production?

timja commented 2 years ago

We would like this so that eventually docker buildx bake support can be added so that we can define one hcl file with all our image builds declaratively configuring things like image tags and args, along with parallel builds out of the box. We use this on linux right now and it works really well.

darkvertex commented 2 years ago

What else?

Off the top of my head, I'd bring up:

If I think of more I'll extend this list. 🤔

FaustinCarter commented 2 years ago

I'd like to add:

darkvertex commented 2 years ago

I remembered another big useful feature: Buildkit has "build-time mounts" support.

Everyone knows about --mount=type=secret to access secret files without baking them in the image layers, but did you know Buildkit also offers:

snorrk commented 2 years ago

Not being able to pass secrets into the Windows container build is painful - for us it means copying down files to the container build host (since that requires a secret) and then having docker slowly discover the container build context and copy the files into the container during the container build (double I/O).

It would be much better to be able to just download the files inside the container as part of the container build.

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.

krokofant commented 2 years ago

Thank you @msftbot for the monthly reminder that nothing has changed. 👀

slonopotamus commented 2 years ago

What do you expect from Microsoft here? You want them to write and submit patches that add Windows support to BuildKit? Why they would do it? Looking at how https://github.com/microsoft/hcsshim/pull/901 is stuck waiting for approval for around a year, it would be more effective to remove Microsoft from dependency chain of this feature.

agowa commented 2 years ago

What do you expect from Microsoft here? You want them to write and submit patches that add Windows support to BuildKit? Why they would do it?

@slonopotamus because of the huge differences in how windows containers and linux containers work. I dought that anyone else has enough knowledge about the internals to do it...

cwilhit commented 2 years ago

Hi all, in effort to reduce spam of the beloved msftbot asking for status ten times over, I am going to freeze conversation on this issue (hoping it pauses the bot spam in the process). Given this issue has over 80 emoji reactions, we hear you loud and clear this is something you as developers would like to see. If and when we have a concrete update to share on our investments in this area, we will reply here and unfreeze the issue.

fady-azmy-msft commented 2 years ago

FYI @muzz

ghost commented 2 years ago

This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.

bplasmeijer commented 2 years ago

Checking the latest status

TBBle commented 2 years ago

Current status on my end:

I do plan to continue trying to get this working, but I'm not going to be able to do much for the next month or two at least, due to personal-life things, e.g., moving home.

I don't know how BuildKit integrates into docker buildx, so it's possible that some of the above could be shortcut by the upcoming Docker 22.06 release's support for containerd on Windows. However I suspect that at the very least https://github.com/microsoft/hcsshim/pull/901 would need to be completed and integrated into Docker's storage system; possibly it'd also need https://github.com/containerd/containerd/pull/4419 since parts of that are directly compiled into BuildKit.

bplasmeijer commented 1 year ago

I dump again into the deep here. where are we with containerd and building windows containers with podman, nerdctl? cc: @jsturtevant

jsturtevant commented 1 year ago

nerdctl uses build kit. Buildkit support is still in progress, you can see the relevant PRs in https://github.com/microsoft/Windows-Containers/issues/34#issuecomment-1235634370.

I am not sure how podman builds containers.

bplasmeijer commented 1 year ago

@TBBle any update from your side? I am checking myself now.

TBBle commented 1 year ago

Nothing from me personally since September 3rd, I'm still on sabbatical. One or two other people have been pushing at the same and related PRs in my absence, but I can't report any measurable progress here.

bplasmeijer commented 1 year ago

Nothing from me personally since September 3rd, I'm still on sabbatical. One or two other people have been pushing at the same and related PRs in my absence, but I can't report any measurable progress here.

Thanks, keep enjoying your ooo.