Support running Windows Containers on linux

[seanyoung]

I realise this might be out of scope but I thought I'd raise it so it can be considered.

Windows Containers are slightly modified docker containers, which contain windows executables and the windows registry. These file systems could be mounted with case-insensitive options, and then a very thin wine shim could run the actual binaries. From what I can see, all the necessary executables and dlls are included in the images (e.g. cmd.exe).

https://docs.microsoft.com/en-us/virtualization/windowscontainers/

This would be super-useful to be able to run e.g. windows builds on linux, see and example of a windows dockerfile here: https://github.com/hyperledger/solang/blob/792df33e712f1142da515b7b69f4a4fe7965b5e4/build/build-llvm-windows.dockerfile

[flouthoc]

@seanyoung Also the idea is pretty much same cause even wine works on linux if we are good to adding another extra layer in image which similar to what we do for .NET runtime.

[seanyoung]

@seanyoung It will be much easier to do similar loading with wine shared libraries as well but again wine has a dependency on huge set of shared libaries. But the idea of above PR is to run executables without runtime layer in container image but will be only scoped to .NET.

Ok.

Edit: I am hoping a similar bind mount for all the shared libaries for wine should make it work just like we are doing for mono. Since static building of wine is not at all easy. https://forum.winehq.org/viewtopic.php?t=31388

Windows Containers come with all the dlls included. It possible that all that is needed is an pe exe loader and the rest of the windows system calls, and the rest of wine is not needed.

[flouthoc]

Windows Containers come with all the dlls included. It possible that all that is needed is an pe exe loader and the rest of the windows system calls, and the rest of wine is not needed.

Could be but wine loads some of its own stuff dynamically as well. I guess that's why people are exploring static build for wine but no one was able to get it working i guess. Once I complete the mono poc I guess i should be able to get more idea.

[rhatdan]

The other big issue is file system issues. You have to have the file that stores your images support treating files with different cases the same. Basically running in Windows mode. Even if we have this, I think volumes might end up being an issue for the same reason. BTW I would love to have something like this, but we could use community help.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[apollocreed]

Hello ! It's a related issue, I'm trying to build a windows container with podman on WSL2 without success. I've tried both ways.

podman build  --platform windows/amd64 -t dockeragent:latest  .
podman build --os windows -t dockeragent:latest  .

my Dockerfile:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

WORKDIR /azp

COPY start.ps1 .

CMD powershell .\start.ps1

here the error

Error: error creating build container: writing blob: adding layer with blob "sha256:8b534d64a7c337eb8a23b425e4f598cd3e7407ddf8c7b2f714d1e7f7ed6a04be": Error processing tar file(exit status 1): link /Files/Program Files/common files/Microsoft Shared/Ink/en-US/micaut.dll.mui /Files/Program Files (x86)/common files/Microsoft Shared/ink/en-US/micaut.dll.mui: no such file or directory

Seems that a DLL is missing

my env

uname -a
Linux  5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cheers

[bb010g]

Is this also the issue for running Windows containers on Windows, and not just running Windows containers on Linux?

[flouthoc]

@bb010g I think this issues is more of a spike for running windows containers on linux and not windows containers on windows not sure but I don't think we have any open issue for windows container on windows.

@apollocreed For building windows containers we have a issue opened at buildah https://github.com/containers/podman/issues/8136#issuecomment-1142194900 and not yet merged PR on buildah ( working for ADD cases ) https://github.com/containers/buildah/pull/4053

[rhatdan]

Currently there is no effort to run windows on windows, and this would definitely need to be done by Windows Engineers (contributors) None of the core developers work in this area.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

There seems to be a slight effort toward this goal going on. Some of the effort could be helped by the freebsd effort ongoing.

[lippertmarkus]

@rhatdan do you have a link regarding the ongoing effort?

[rhatdan]

https://github.com/containers/podman/pull/15372 https://github.com/containers/podman/pulls?q=is%3Aopen+is%3Apr+label%3Awindows

[lippertmarkus]

@rhatdan sorry, but can you clarify? Most issues seem to be around running a podman machine in WSL and only using the CLI on Windows. This issue here is about running Windows containers on a Linux host by having e.g. a wine shim, so completely different, or am I missing something?

[rhatdan]

Sure, the basic premise is getting the podman commands easier to run on Windows. The difficult part would be getting containers/storge to work, since I don't believe Windows has any support for layered file systems. But baby steps.

As far as running Windows containers on a Linux Box, the biggest problem is handling the NON-mixed case file system easily. Linux has this feature, but setting up the file system is not easy. You would need to split storage for linux containers from Windows containers, onto two separate partitions, and no one has worked on it. Then getting an OCI Runtime with crun wrapping WINE. Bottom line is the core maintainers have not worked on this, mainly because of time. If outside contributors worked on it, it would move ahead faster.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[Hi-Angel]

this issue had no activity for 30 days

Here's a ⁺¹ comment to create activity. That could be another big advantage of Podman over Docker.

[a-clean-kitchen]

Commenting to show interest 👍 and keep this issue alive!

[naitmare01]

Dropping a comment to show interest!

[rhatdan]

There is discussion on this happening in crun. https://github.com/containers/crun/issues/1178

[wbehrens-on-gh]

Commenting to show interest

[frankbou]

Commenting to show interest

[vrothberg]

I honestly do not see how we could support Windows Containers on Linux. We do not have Windows expertise, and once merged we have to maintain it and give upstream support.

I really like the idea though and find it cool but I think that shipping such a feature is just not sustainable for upstream.

[bhack]

I honestly do not see how we could support Windows Containers on Linux. We do not have Windows expertise, and once merged we have to maintain it and give upstream support.

I really like the idea though and find it cool but I think that shipping such a feature is just not sustainable for upstream.

True, but have you explored a maintaiener partnership with the Wine ecosystem?

[rhatdan]

I agree we don't have people to support this in the RHEL sense, but if we could get Wine experts to contribute and work with this, I think it would be a nice Upstream feature for Podman.

[vrothberg]

My comment above focused on upstream only. I don't see how we can even do community support without any expertise. I personally prefer to not ship a feature upstream rather than give the impression that incoming issues can be triaged or get any priority.

In other words: I don't think we have time for it.

[rhatdan]

The only way to build expertise is to experiment. We don't have great expertise in WASM and yet crun-wasm exists. If someone steps up to do the work of getting WINE to work. we would expect that person to help with support of Windows clients. If such person disappears we can slowly deprecate the support. If the idea has momentum someone will step up to support it.

[seanyoung]

I've spent a little time looking at how windows containers work. There are two modes Windows containers can be run in, using the windows docker command line tools it looks like this:

docker container run --rm -it --isolation=hyperv mcr.microsoft.com/windows/server:ltsc2022

or

docker container run --rm -it --isolation=process mcr.microsoft.com/windows/server:ltsc2022

In process mode, the container runs like you would expect: the windows kernel services system calls from the container directly. Emulating this mode requires implementing Windows system calls or using wine.

The Hyper-V mode is interesting. There a small "utility VM" which runs in Hyper-V (a hypervisor). In the docker container there is a UtilityVM directory that contains a bunch of UEFI files that make up an OS. If I am reading the docker source code directly, both the UtilityVM and the docker container files are presented to the VM as a vhdx file. I guess we would need to figure out how to boot this VM and provide it with the disks it needs (I doubt that uses virtio based disks!).

https://forums.docker.com/t/use-of-container-utility-vm-folder-docker/118403/6 https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container https://learn.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-2022%2Cwindows-11

[bhack]

So does it means that it could theoretically boot with qemu hyperv emulation? https://www.qemu.org/docs/master/system/i386/hyperv.html

[seanyoung]

So does it means that it could theoretically boot with qemu hyperv emulation? https://www.qemu.org/docs/master/system/i386/hyperv.html

I suspect so. Probably require some debugging to figure out exactly how to set things up.

[JohnnyElvis]

Are there any topics where a person without the ability to read code could help?

[Hi-Angel]

It seems, implementing Windows containers could fill a market with no competitors. Quoting a bit of discussion from kde-devel

For Windows however, Microsoft has limited the container stack to not allow anything GUI related to work. The underlying libraries may be there, but the equivalent display server components are not operational.

To complicate things further, on Windows certain permissions are restricted to the interactive console and are not possible to do as either a scheduled task or as a system service. Usage of existing Windows automation frameworks such as Powershell Remoting or SSH will therefore not work if we want things to perfectly replicate a end user environment - because those will run the command(s) as part of a non-interactive session (even if the user we connect as is the same one logged in on the desktop console).

If Podman could run Windows containers, there would be no reason to adhere to such arbitrary limitations. Which would make it the preferred (the only one really) choice for a number of tasks.

[vrapolinario]

Hello folks, MSFT person here - first, wanted to make myself available to answer questions on Windows containers. Keep in mind I have close to zero knowledge on Linux. I focus on Windows containers, Kubernetes, Docker, etc. but everything running on the Windows OS (aside from the needed K8s control plane stuff).

I understand the desire to run Windows containers on Linux, so let me just put something out there: While that might be possible with Wine (or whatever method) that would not be supported by Microsoft. Of course, something working and being supported are different things and if this works, I'm sure some users will end up using it - I'm more just making this clear than anything else.

The other thing I wanted to understand is: Wouldn't be easier to actually have Podman support Windows containers on Windows? Today, docker, moby, and containerd interact with Windows (for Windows containers) via HCS. https://learn.microsoft.com/en-us/virtualization/api/hcs/overview I might be wrong, but I assume there could be some translation of container operation for Linux to HCS. I'm not saying it would be easy, but most likely easier than the running Windows containers on Linux? Maybe even take this discussion in parallel?

[Hi-Angel]

The other thing I wanted to understand is: Wouldn't be easier to actually have Podman support Windows containers on Windows?

Isn't it a supported case?

That said, I'm not sure I get your point: do you want a Linux running a Windows VM running a Windows container or how do you want that to work on Linux? And even if so, that still requires buying a license, does it not…? I don't see how that would solve the case.

[1480c1]

Seems the idea of windows containers on windows was mentioned before https://github.com/containers/podman/issues/11809

[vrapolinario]

AFAIK Podman doesn't support Windows containers, just Linux containers. Did I miss something?

do you want a Linux running a Windows VM running a Windows container or how do you want that to work on Linux?

Well, the idea would be for Podman to run on Windows - if that's on a Windows host directly or Linux running a Windows VM, it wouldn't make difference.

And even if so, that still requires buying a license, does it not…? I don't see how that would solve the case.

Well, Windows containers need to be license anyway. If you manage to run a Windows container on Linux machine, you are using a Windows OS, so that would have to be licensed anyway.

[vrapolinario]

@1480c1 Thanks for noting that. That's unfortunate. I could see huge advantages to having Podman support for Windows containers.

[Hi-Angel]

do you want a Linux running a Windows VM running a Windows container or how do you want that to work on Linux?

Well, the idea would be for Podman to run on Windows - if that's on a Windows host directly or Linux running a Windows VM, it wouldn't make difference.

I think there's a confusion. The issue is about Linux host to run Windows container. E.g. for CI purposes or for testing purposes, or just to run some service… So I am trying to figure out how your suggestion fits into a solution, and running a whole VM so far the only one I see. If that's indeed your suggestion, then needless to say that sounds very heavyweight.

And even if so, that still requires buying a license, does it not…? I don't see how that would solve the case.

Well, Windows containers need to be license anyway. If you manage to run a Windows container on Linux machine, you are using a Windows OS, so that would have to be licensed anyway.

No, people have been running Windows apps on Mac OS and Linux via WINE with no license required for ages. Admittedly, Idk what a Windows container contains, but I've never heard of anyone requiring to pay to launch a container.

[vrapolinario]

I think there's a confusion. The issue is about Linux host to run Windows container.

No, I get that. I was proposing a different approach, which based on the comment above has been rejected. I was proposing that Podman could run Windows containers on Windows - they can still be in a CI/CD pipeline, just a different OS. I get this is not the original proposal, but based on the comments I felt this was becoming a harder task than anticipated, hence my suggestion.

No, people have been running Windows apps on Mac OS and Linux via WINE with no license required for ages. Admittedly, Idk what a Windows container contains, but I've never heard of anyone requiring to pay to launch a container.

That's the thing. If you get an MSI/EXE package that was compiled for Windows and run on Linux, (or MacOS) that's totally fine. You are not using the Windows binaries, but simply translating the APIs. Now, if you run a Windows container, then you have Windows binaries, which need to be licensed. Don't shoot the messenger 😊.

[vrapolinario]

For more information on licensing for Windows containers: https://learn.microsoft.com/en-us/virtualization/windowscontainers/images-eula

[Frankkkkk]

For more information on licensing for Windows containers: https://learn.microsoft.com/en-us/virtualization/windowscontainers/images-eula

That's for the windows base image. Not all windows containers use these base images, isn't that so ?

[vrapolinario]

Yes, they do. All Windows images use the Windows containers base images from MCR. You might have to track the original base image if you are using .Net for example, but they all do use the base images. If it is a Windows containers, there's no escape from that.

[Hi-Angel]

I was proposing that Podman could run Windows containers on Windows - they can still be in a CI/CD pipeline, just a different OS.

This "different OS" isn't hackable (due lack of source code), which isn't something developers are typically interested in. So I don't see your suggestion as a viable solution. Even if what you say about the license is true, well… at least the amount of "non-debuggable opaque code" would be reduced as much as possible when there's no actual Windows host around the container.

[Jackenmen]

@vrapolinario If there's interest from Microsoft to champion the effort of Podman support for Windows containers on Windows then the relevant issue is #11809 and since it's locked, a new one would have to be created. The conclusion on that issue was that the Podman community needs Windows engineers to allow Podman to support this. Making yourself available to help with this is a good first step but I imagine it may not be enough unless some volunteer jumps in so MSFT may just need to help implement this themselves. Either way, a new issue mentioning #11809 would be a good way to move this forward since this issue is about a different (likely harder to implement and unlikely to be of Microsoft's interest) feature of supporting Windows containers on Linux (where IMO the main benefit is that it requires A LOT less resources than running a full Windows VM with a Docker/Podman daemon that you can connect to from the Linux host it's running on).

[vrapolinario]

I can't (and won't) debate that. I didn't mean to stear off the original conversation. Was just proposing an alternative which would help Windows apps be used with Podman. Honestly, I'd love to be able to tell Microsoft and Windows customers that they have Podman as an option to run Windows containers.

In any case, I'm here if you have question on Windows containers and I can help.

And please, keep in mind the licensing. It's not if it's true. If it is a Windows container it requires a Windows license.

[vrapolinario]

@vrapolinario If there's interest from Microsoft to champion the effort of Podman support for Windows containers on Windows then the relevant issue is #11809 and since it's locked, a new one would have to be created. The conclusion on that issue was that the Podman community needs Windows engineers to allow Podman to support this. Making yourself available to help with this is a good first step but I imagine it may not be enough unless some volunteer jumps in so MSFT may just need to help implement this themselves. Either way, a new issue mentioning #11809 would be a good way to move this forward since this issue is about a different (likely harder to implement and unlikely to be of Microsoft's interest) feature of supporting Windows containers on Linux (where IMO the main benefit is that it requires A LOT less resources than running a full Windows VM with a Docker/Podman daemon that you can connect to from the Linux host it's running on).

Sounds good. I'll take this internally, but even if there is interest, it won't happen so soon.

[geoffreysmith]

I don’t remember my original intent in starting the question but our issue has actually become worse and/or changed but we run “atomic desktops” on development machines for a number of reasons which usually use has a development workflow of toolbox/distrobox for sandboxed dev then eventually on commit whether or not the build is ready to product gets sent to a build matrix where more or less it gets turned turned into a lightweight K8S container that way we’re building to prod always and release on git tags. We do bypass the kernel but not often usually for performance reasons. Docker has come a long way in running rootless and/or reasons we began using CRIO.

So we use our host for OSX/Windows and ssh into a big SSH dev instance where we spend most our day using VSCode or VIM over SSH to big Debian boxes. WSL won’t work as we have auditing software (ZScalar) that prevents loopback and we do sometimes need to modify the Linux kernel for performance reasons very specific to us… but in any case 99% of our day is on a giant server and it’s much easier to have our host machine with Teams/etc and get good driver support for things like power management that only a few people will do the Debian laptop and VM into Windows thing for work communication.

But the problem we have now might be different. While our use of multi cluster setups is to support internal business apps and deal with large data intake, workflow and light on-prem ML. The rest of the company isn’t probably isn’t even aware and just knows their data is in OneDrive and that Office 365 just works. We will plug into the graph API but that’s simply to update reports, etc. which can be done entirely in Linux.

But that’s to just give you an overview. Our problem now is that it is trivial to output the results of 15tb of data process from multiple sources and drop it into essentially a certain teams project folder or watch when files change, etc. PowerPoint is presenting different challenges. This is definitely an edge use case but we can take the OpenXML spec , extract it have a generator that creates go code then “correctly” create guidelines, placeholders, etc from a designer who might’ve taken shortcuts. Then we have a content db where we store assets like logos and variables when we “know” certain things about the project. We found creating things from master templates/layouts/placeholders and not avoiding them even if we programmatically enter data avoids a lot of issues. We give then have a word document (user preference) where they enter data where we can’t fill out and essentially it creates a json schema we can validate and enter into text boxes etc. The point being is that PowerPoint does things like vertical-spacing poorly compared when you have text titles that might overflow to three lines to stay within a column layout and the description box will not adjust to 0.5” from the bottom. Same with grid lines where we can snap to but the user won’t or forget that 5 minutes before a meeting. We are trying to treat this essentially like a website with components that are aware of other components and adjust accordingly. Even worse is grouping or like adding a slide with project team members. Adding one at the last minute that can’t be validated by the server means we need a few simple plugins. Plus we also have the ability to have them mark items as basically “I want to not to stick to the design guide.”

We’d really like the ability to test this like we do Linux microservices. Create reasonable build matrix of Windows versions and PowerPoint versions and create test slides. Our syntax is usually correct but typography on Windows and other display issues means we need real copies, create a few test slides and make sure PowerPoint or whatever is looking right. Then run through them and basically create a selenium/web screenshot type setup so we can make sure we aren’t doing any regression or if a designer gives us a new slide what worked on their machine also works on lower DPI screens, different versions etc. what we are doing is simple and our validating the schema is correct we’d like to boot up a lightweight container, open a test file go through the slides take screenshots and make sure we past tests. Accidentally moving a dynamically placed text box off grid isn’t the end of the world and we aren’t looking for pixel perfect. But right now we’d have to build ansible VMs of giant Windows images in HDR, several different resolutions last x versions and then multiply that by the last couple versions of PowerPoints.

Containers let us build caches and we can often debloat them. We are a large >50k org and have whatever high level of partnership so licensing or whatever isn’t an issue I’m sure. It’s a pretty simple ask in the end “do our plugins or server side processing doing the same thing work” and while we do this with Linux microservices on every build there’s a huge difference between a 300 mb image with build matrix and cache is huge. Not to mention we use GitHub on-prem to push images up with the runner and some minor custom tooling and it “just works” with GitOps. We don’t need this to create an image though catching an error and rolling back the services and plugin package that power this would be nice.

So oddly while we do a lot of complex things we can call the Office365 API to make sure more or less the API didn’t change (some traditional sysadmin or secops person didn’t change permissions randomly), there’s no visual or Windows element to it so far as needing to not boot Windows.

What I’d rather not do is create some strange VM workflow maybe with snapshots to speed it up and create an entirely new runner, etc for one sort of minor thing.

So really any container that could slim Windows down and/or containerize it would be helpful. Not to kiss ass but the open nature of Azure/Office 365 (even though we do our K8S on-prem means we build out our apps and call out to a simple API means we actually kept to Microsoft).

Surely there’s a way internally manual apps are tested for graphical regression. I get that might not be available to us but the UI rendering in Windows differs so much we can’t reliably test it and I’m sure it’s due to years of bowl level crust built up over the years. Even if we don’t have a traditional UI interface and could only take screen shots (using ansible or some scripting to open a file, create a screenshot, move to next template, etc) we can create a Windows host cluster and deploy containers on them. Maybe calling DirectX (or whatever) but I’m not familiar enough with creating graphical apps to know how the rendering engine works. If we generate a UI it’ll be a website but we are mostly Rust/Golang/C++ programmers, so as long as we can do something to record output from the CLI that’s what we are looking for.

Sorry for the wall of text and maybe getting off topic but essentially we’re looking to generate screenshots from a container (docker or otherwise) even if it means layering on packages not included in the Windows container, Podman or Docker.

I’m sure internally you might literally have bare metal racks that not only have full Windows installs but also different physical graphics cards. But also your day to day devs probably have a slimmed down solution that works 90% of the time.

Sorry for the wall of text but thinking this through a bit but I understand not wanting an interactive GUI for licensing issues on containers, but even if we have to call low level com interops to open up an app, open a file, move through slides, take screenshots that might actually be more automated and a better solution.

Let me know if you have any ideas. Thanks!

[branpurn]

Sorry for the wall of text but thinking this through a bit but I understand not wanting an interactive GUI for licensing issues on containers, but even if we have to call low level com interops to open up an app, open a file, move through slides, take screenshots that might actually be more automated and a better solution.

Let me know if you have any ideas. Thanks!

Had Windows Server Nano not become vaporware, it was probably intended to evolve for use cases such as these

[vrapolinario]

Had Windows Server Nano not become vaporware, it was probably intended to evolve for use cases such as these

Not exactly. Nano Server was never intended to support legacy apps, so this would not be a supported scenario for that base image. With that said, Let me clarify that we have the following images: Nano Server: Slim image that needs to have the framework target this Windows OS type. Very little set of APIs available, hence the framework needing to target it. Supports .Net (not .Net framework) and 3rd party frameworks (Java, etc.) as long as they target Nano. Current downloaded size is ~290Mb. Server Core: Medium size. Focused on legacy apps. Supports .Net Framework (and .Net) but in order for that to work, has many more APIs available resulting in a larger image size. Keep in mind this image should never be compared to Linux as it's not a use case for new apps - but it's much better than a VM. Current downloaded size is ~5.4Gb Server: Full set of Windows APIs, includes support for GPU, and the Windows GUI APIs. Keep in mind it doesn't give you a GUI or RDP, but the GUI API is there, so this image can be used for automated UI testing (as mentioned by @geoffreysmith). Current downloaded size is ~11.9Gb.

I know you all see this as a monstrous container in terms of size. Keep in mind the goal here is not to compare to any Linux image, but be an advantage versus running the same on a VM. A container with that size, but that can fit into a modern stack is still better than a VM you need to manage the old way.

Now, the problem here is running Office apps (Word, PowerPoint) on Windows containers. Currently, they are not supported. I'm not sure if they work on the Server image, you might want to try it. The image you might want to try is: mcr.microsoft.com/windows/server:ltsc2022