search

containerd / nerdctl

contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...
Apache License 2.0
8.16k stars 609 forks source link

Windows support #28

Open awakecoding opened 3 years ago

awakecoding commented 3 years ago

I am looking for a way to use containerd on Windows Server 2019 with a Docker CLI replacement like this one. I see there are Linux builds, but has a Windows port been attempted?

Long story short, containerd for Windows should handle graceful shutdowns for containers, but the current version of Docker is still using HCS v1 + some really broken stuff under the hood, meaning Windows containers have never supported graceful shutdowns (https://github.com/microsoft/Windows-Containers/issues/16).

There is no clear migration path for Docker to adopt containerd on Windows in the short term, so I would be fine with installing containerd on Windows (https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-containerd) but then I just need a proper CLI on top of it, other than 'ctr' which is very primitive and marked as unsupported/unstable.

I am not familiar with Go, but how difficult would it be to attempt a Windows port of this tool?

AkihiroSuda commented 3 years ago

Contribution is wanted 😃

jsturtevant commented 3 years ago

I took a quick look and it would be doable but would require some work, there are assumption that this would run on linux like (there are probably more):

https://github.com/containerd/nerdctl/blob/06f3a7c8913abfae11095ee0ebbba0f013cc3f6f/exec.go#L229

https://github.com/containerd/nerdctl/blob/435f5331df3208b9e297290b4e8bec4fe2bd4a2e/run.go#L762-L766

https://github.com/containerd/nerdctl/blob/435f5331df3208b9e297290b4e8bec4fe2bd4a2e/pkg/lockutil/lockutil.go#L33

https://github.com/containerd/nerdctl/blob/435f5331df3208b9e297290b4e8bec4fe2bd4a2e/pkg/containerinspector/containerinspector.go#L73

jsturtevant commented 3 years ago

I got started on this. Will likely want to break this down into many different PRs

AkihiroSuda commented 3 years ago

Thanks!

jsturtevant commented 3 years ago

Got it to build and connect 🚀 

PS C:\projects\nerdctl\_output> .\nerdctl.exe ps -a
CONTAINER ID    IMAGE                                        COMMAND                   CREATED          STATUS                   PORTS    NAMES
multi-arch      k8s.gcr.io/pause:3.4.1                       "/pause.exe"              3 days ago       Created
multi-arch2     k8s.gcr.io/pause:3.4.1                       "/pause.exe"              8 minutes ago    Created
multi-arch3     k8s.gcr.io/pause:3.4.1                       "/pause.exe"              8 minutes ago    Up
test            mcr.microsoft.com/windows/nanoserver:20H2    "echo hello"              9 days ago       Exited (1) 9 days ago
test-1          mcr.microsoft.com/windows/nanoserver:20H2    "echo c:\\License.txt"    9 days ago       Exited (1) 9 days ago
test2           k8s.gcr.io/pause:3.4.1                       "/pause.exe"              7 days ago       Created

Needs some more work to be able to run containers

jsturtevant commented 3 years ago

I was able to get nerdctl run to work! But I did run into issues running multi-arch docker manifests (https://github.com/containerd/containerd/issues/5297) which should be resolved by https://github.com/containerd/containerd/pull/5298. 

.\nerdctl.exe run --name nerdclttest1 mcr.microsoft.com/windows/nanoserver:20H2
Microsoft Windows [Version 10.0.19042.867] 
(c) 2019 Microsoft Corporation. All rights reserved.                                                                                                                                                                                            C:\Windows\system32>

Although it is rough, I opened #164 just to show the work involved. I would propose breaking this down into at least two PRs for easier reviewing:

  1. split out the Logic that is tied to Linux to their separate files so that we can build the windows binary. This would allow producing and .exe but functionality would be limited. From my testing nerdctl ps would work and maybe some other basic commands.
  2. Add in run support which requires a bit more refactoring.
  3. After that we can approach each other functionality iteratively from there.

What do you think @AkihiroSuda?

AkihiroSuda commented 3 years ago

SGTM, thanks

awakecoding commented 3 years ago

I am revisiting this ticket since Rancher Desktop 0.5 just added initial nerdctl support. My understanding is that the latest version of nerdctl doesn't support Windows containers, unless one builds a branch from @jsturtevant from source + installs containerd.exe manually?

AkihiroSuda commented 3 years ago

I am revisiting this ticket since Rancher Desktop 0.5 just added initial nerdctl support. My understanding is that the latest version of nerdctl doesn't support Windows containers, unless one builds a branch from @jsturtevant from source + installs containerd.exe manually?

Yes. Help wanted to move https://github.com/containerd/nerdctl/pull/197 forward

jsturtevant commented 2 years ago

should we close this? I've opened a few other issues to track the next steps in windows support, otherwise this could be used as an epic?

jsturtevant commented 2 years ago

https://github.com/containerd/nerdctl/issues?q=is%3Aopen+is%3Aissue+label%3Aplatform%2FWindows

AkihiroSuda commented 2 years ago

I think we can keep opening this

brwilkinson commented 1 year ago

I was wondering what the current status is for Windows support ?

Based on below, can I assume this issue can be closed now?

Was there anything in particular outstanding?

TBBle commented 1 year ago

https://github.com/containerd/nerdctl/issues?q=is%3Aissue+label%3Aplatform%2FWindows%2FNon-WSL2+is%3Aopen are the outstanding issues. (Same list as https://github.com/containerd/nerdctl/issues/28#issuecomment-976837413, but the label has been renamed since that comment was made.)

I have no strong opinion on whether we should keep this issue open or not. This issue does appear to lack clear (unmet) success criteria though.

AkihiroSuda commented 1 year ago

This issue does appear to lack clear (unmet) success criteria though.

We still need to support builds and compose AFAICS. It would be also nice to have a double clickable “SETUP.EXE” too.

TBBle commented 1 year ago

https://github.com/containerd/nerdctl/issues/627 exists for building, it is currently blocked on https://github.com/microsoft/Windows-Containers/issues/34 and through to buildkit and related. Once we have buildkitd for WCOW stable, hopefully the integration won't be too far from how it's used on non-Windows.

I know approximately nothing about compose, nor the compose support in nerdctl. I don't yet see a ticket for it to work on Windows, so I'm guessing not much time has been spent on that yet, given we're still lacking build support.

Are you thinking of a simple installer for nerdctl.exe, or more like the "Full" binary release, bundling the whole (Windows-supported) stack of dependencies with service setup etc?

Or perhaps something different? We could recreate something like MS's existing containerd/nerdctl script, perhaps.

AkihiroSuda commented 1 year ago

Are you thinking of a simple installer for nerdctl.exe, or more like the "Full" binary release, bundling the whole (Windows-supported) stack of dependencies with service setup etc?

I’m expecting the latter one

SamuelMarks commented 4 months ago

I can help with the installer and porting Go code cross-platforms.

FYI: That linked issue just got an update 9 hours ago:

// Update: Buildkit 0.13.0 is now released with windows binaries - Release v0.13.0 · moby/buildkit (github.com) 🎉 Thank you all who have continued to test the releases, WCOW support is still experimental but we are on the way to stabilization. We will appreciate any feedback; this will aid in our prioritization especially on closing the Linux/Windows feature-parity gap. Feel free to open the issue at moby/buildkit.

// UPDATE: Stabilization efforts still going on. We have had several releases since v0.13.0, now at v0.14.1. Aiming for GA by September 2024. Feel free to give it a try and open any issues on moby/buildkit.

https://github.com/microsoft/Windows-Containers/issues/34#issuecomment-2187976559

TBBle commented 4 months ago

At this point, the hardest remaining thing for "Windows support" is getting compose working, and I say that simply because I have no context on compose, so don't know how far the current code is from "working" if we simply remove any checks for Windows.

That said, building a full-stack installer (containerd, nerdctl, buildkitd, CNI plugins, is there a composed?) is a fiddly task, so having that done even before we have compose support ready would definitely make it easier for people to test and start working with the nerdctl stack. And really, that's more important than closing any specific ticket, I think.

Oh, we need to look closely at the CNI setup, nerdctl generates CNI 1.0.0 configs, but the last-released WinCNI nat plugin can't read those. The WinCNI source has been updated, but no new releases shipped. So we might need to bundle a local build of the WinCNI plugin in such a package. (Also, the generated config might be slightly wrong. I worked through this while testing nerdctl/buildkitd support on Windows, see https://github.com/microsoft/Windows-Containers/issues/34#issuecomment-1820966085 for example. However, I was working from an existing Docker-created NAT network, so that's not necessarily the complete target experience.)

TBBle commented 4 months ago

On the topic of installers, I just came across https://github.com/microsoft/containers-toolkit which appears to be the start of centralising various containerd-nerdctl-buildkitd stack install scripts in a PowerShell module. I haven't tried it, but it might be a reasonable alternative or placeholder for the full setup.exe install stack we've talked about here.

slonopotamus commented 4 months ago

Hmmm, maybe it's time to extend Stevedore a bit 😈

slonopotamus commented 4 months ago

How do you handle the situation with nerdctl requesting CNIVersion=1.0.0 in C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist while CNI plugins only support 0.3.0? There is https://github.com/microsoft/windows-container-networking/pull/101, but it was never released.

UPD: I missed that @TBBle talks about it in https://github.com/containerd/nerdctl/issues/28#issuecomment-2194589804. Maybe we actually should just fork and release the thing while waiting for upstream.

slonopotamus commented 4 months ago

That said, building a full-stack installer (containerd, nerdctl, buildkitd, CNI plugins, is there a composed?) is a fiddly task, so having that done even before we have compose support ready would definitely make it easier for people to test and start working with the nerdctl stack. And really, that's more important than closing any specific ticket, I think.

Look what I've got for you!

image

Basic usage instructions

slonopotamus commented 4 months ago

Maybe we actually should just fork and release WinCNI while waiting for upstream.

And... done!

SamuelMarks commented 4 months ago

@slonopotamus Great to see!

PS: Ping me if you need/want a hand

TBBle commented 4 months ago

A quick note while I think of it: If I remember correctly, Windows only allows one NAT network to be created. So we might want to look at being able to read an existing NAT network config back from HCN/HNS when generating the CNI config, rather than just stamping out a pre-determined config as I recall we do now.

That will make life easier for systems where Docker is also installed, particularly if the user wants Docker to continue to work. Alternatively, matching whatever Docker's defaults are for the NAT network might bridge that gap for the majority of cases; assuming they're not dynamic... and assuming we're not already doing that and my experience wasn't an outlier.

slonopotamus commented 4 months ago

Windows only allows one NAT network to be created

I'm not sure about that:

image

slonopotamus commented 3 months ago

Well... Things are not fully functional. nerdctl container gets default gateway from C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist but all the rest network settings come from pre-existing nat network.

>nerdctl run --rm -it mcr.microsoft.com/windows/nanoserver:ltsc2022 ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : minwinpc
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

Ethernet adapter vEthernet (27157d866c9ea519d931a2cf112db5037656d84b7f81ed7d160a7dd00b084a89_nat):

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Container Adapter
   Physical Address. . . . . . . . . : 00-15-5D-16-73-AA
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::fa0b:424f:9b9:a66b%60(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.20.60.34(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 10.4.0.1
   DNS Servers . . . . . . . . . . . : 192.168.0.1
   NetBIOS over Tcpip. . . . . . . . : Disabled

Interestingly, neither nerdctl- nor docker-based container can access IPv6 address.

UPD: Okay, I can force nerdctl to create a separate NAT network by adjusting name in C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist. And IPv4 + DNS starts to work in it (IPv6 still doesn't, but that's a separate story). The caveat is that I now need to pass --network <network_name> to each nerdctl run.

Can anyone explain what's the purpose of nerdctl/default-network label in C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist? It seems to be ignored, nerdctl run just defaults to nat: https://github.com/containerd/nerdctl/blob/v2.0.0-rc.0/cmd/nerdctl/container_run.go#L109-L116

TBBle commented 3 months ago

I must have misremembered about the NAT network limitation then, sorry. I'm sure there was some limitation, but maybe it was in Docker or something? (It may have been in the older virtual network API, too.)

nerdctl container gets default gateway from C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist but all the rest network settings come from pre-existing nat network.

This is a known behaviour issue to be fixed in the WinCNI plugins: If you name an existing network, it uses it as-is, it does not update it to match the CNI configuration. (The default gateway lives on the net connection to the network, not the network itself).

See https://github.com/microsoft/windows-container-networking/issues/98

There's also an issue around the WinCNI plugins faultily matching their network binary to their type rather than their name, when they are creating a network: https://github.com/microsoft/windows-container-networking/issues/57 It's possible this is why I recalled multiple NAT networks being a problem...

slonopotamus commented 3 months ago

There's also an issue around the WinCNI plugins faultily matching their network binary to their type rather than their name, when they are creating a network: https://github.com/microsoft/windows-container-networking/issues/57

I believe that issue is outdated. I was able to use a different network name in nerdctl-nat.conflist. It properly created a separate nat network and used it via nerdctl run --network my_awesome_nat. The nuisance is that I have to say --network my_awesome_nat because nerdctl run doesn't lookup network name with nerdctl/default-network label and blindly uses nat, as I said before.

https://github.com/microsoft/windows-container-networking/issues/98

I'm not sure it is a good idea. It will break things. I'd solve the issue from a completely different angle. If a network is already created, why user needs to write config file at all? Just inspect network and use settings from it.

slonopotamus commented 3 months ago

BTW, there is a new release of win-cni: https://github.com/microsoft/windows-container-networking/releases/tag/v0.3.1