Support Linux images in Docker for Windows

schmichael commented 7 years ago

Nomad only supports Docker for Windows 17.03.1 in Windows Container mode. Attempting to run Linux containers will fail because Nomad will create Windows-style volume bind paths instead of Linux-style.

We should:

[ ] Detect Docker for Windows mode in the fingerprinter
- [x] Requires updating go-dockerclient to support the undocumented Isolation key in the /info API - fsouza/go-dockerclient#649
- Isolation: "hyperv" means Windows mode
- Isolation: "" means Linux mode
- [x] Add new attr.driver.docker.os_type to expose Docker mode to constraints
[x] Create volume paths based on the image's OS
- [x] Requires updating go-dockerclient to support the 1.29 InspectImage API - fsouza/go-dockerclient#649
[ ] Adjust signal handling to be based on the runtime os, rather than the nomad-client os.

Stretch goal:

[ ] Vary the output of nomad init based on Docker for Windows mode (redis:3.2 for Linux, redis:nanoserver for Windows)

tomqwpl commented 3 years ago

I'm also very interested in this functionality. I'm sure my set up is common, Windows 10 running Docker Desktop in WLS2 mode. I've used docker on Windows for some time and have never, ever run a Windows docker image, I've always run docker in Linux mode, scheduling linux containers. For me it's kind of surprising that nomad doesn't just support this, naively I assumed nomad just talked to the local docker daemon to schedule containers.

tgross commented 3 years ago

For me it's kind of surprising that nomad doesn't just support this, naively I assumed nomad just talked to the local docker daemon to schedule containers.

As mentioned several times in this issue, Nomad is shouldn't need any particular support for launching Linux containers on Docker on Windows. It just hasn't been verified. There are lots of weird quirks to the WSL2 environment (ex. a monolithic kernel without modules we were at one point using to detect networking isolation support because that's how virtually every other Linux distro builds their kernel out). So we do want to make sure we have verifiable support for Windows Server 2019 environments. Windows 10 desktops, however, are not a production environment so it's just not likely to ever be a test target for us.

tomqwpl commented 3 years ago

@tgross So if I ignore the warning and try to schedule a linux container anyway, it should work? I've read through the issue but been confused by the status of this.

lutetium174 commented 3 years ago

@tomqwpl Not in my experience. The problem I've found is that the docker driver is marked unhealthy and any attempted use of the driver in my jobs fails. You would need to switch to Windows containers.

tomqwpl commented 3 years ago

@lutetium174 Right, that was mind of my impression too. I think hence my confusion here. There seemed to be talk earlier of it just working at some point, but it certainly doesn't appear to now for that reason.

gaetandezeiraud commented 3 years ago

I'm also very interested. You would need to switch to Windows containers. Don't work for me. Need Windows 10 Pro maybe? (I'm on Home on my new laptop) @lutetium174

dg-eparizzi commented 3 years ago

We have a solution that's a mix of dockerized services (Linux) and some native .NET framework apps. I was under the impression that Nomad would be able to orchestrate both types of jobs without issue but I'm seeing Linux containers are not supported on Windows hosts? Linux containers have been supported in Docker for Windows for a while now so I don't understand why is this an issue for Nomad.

Can someone clarify whether this is a potential approach? We're using Windows 10 Enterprise 2019 LTSC, so we're not even running Docker using WSL2, but Hyper-V instead...

gthieleb commented 3 years ago

I want to showcase to windows guys how easy it is to get started with nomad together with docker to run some linux container workload on a windows box.

Installation of consul / nomad is very easy in windows and in this showcase I'd like to have both of them running in dev mode:

nomad agent -dev
consul agent -dev

Furthermore I'd like to use docker desktop (wsl2 engine) and run nomad/consul in dev mode. But starting a nomad job apparently fails due to the error message of the driver:

Can you shed some light on why this should not be supported/not work? With WSL2 I don't have any chance to either start docker natively or register as a (dedicated linux) nomad client in consul.

das974 commented 3 years ago

I has a similar need for showcasing Windows with the WSL 2 Docker backend. I worked around this by running the server in WSL2 and a client on the Windows host. This allows me to run .Net apps on the windows host and Linux Docker containers in the WSL backend.

You basically end up with 2 nodes with the same name.

Linux Nomad Server

Windows Nomad Client

Enabled only raw_exec for Windows

VSCode Tasks For Reference

This assumes you have downloaded nomad and is in the path (I use a separate task for this)

{
    "label": "Nomad WSL2",
    "type": "shell",
    "command": "bash",
    "args": [
        "-c",
        "sudo nomad agent -config=./nomad/linux/config -bind 0.0.0.0"
    ]
},
{
    "label": "Nomad Windows Client",
    "type": "shell",
    "command": "nomad",
    "args": [
        "agent",
        "-config=${workspaceFolder}\\nomad\\windows\\config"
    ]
}

qagwaai commented 3 years ago

any updates on this?

FibreFoX commented 2 years ago

I find it a bit frustrating, getting excited from the first blog-articles and tutorials about Nomad, which itself proclaimes to provide an easy developer experience, just to find out that this experience is "just" for Linux-based and MacOS(X) developer systems. Myself I am mostly using Windows as my daily driver, and having a mixed type of systems among our developers, I would like to have a shared configuration in our projects.

As "Docker-for-Windows" itself is provided in multiple ways (via HyperV or WSL) there probably is "just" a complexity-issue of the driver. I am confused why this is still an active issue at least for HyperV based Linux-containers.

Is there any plan to provide a workaround to this? Like "just" connecting to a docker-socket or remote-port?

I will look forward to use Nomad (mostly because I dislike the whole Kubernetes ecosystem due to its complexity and giant fragmenting YAML files), but will have to create separate VM for it. Or is there a "simple solution" via Vagrant? (I mean, its a HashiCorp-product, maybe the chances are good, hehe)

EDIT: ... RTFM ... there are instructions for a local Vagrantfile setup: https://www.nomadproject.io/docs/install/quickstart#run-nomad-in-vagrant ... sorry for the noise, but it would be cool if it could work without Vagrant (I think this is nice, but only a workaround).

agajdosi commented 1 year ago

This is very disappointing for such a beautiful product.

Seikilos commented 11 months ago

I also was happy to test drive it, only to find out, that docker on windows with linux containers does not work. Since I am building linux docker images and use them, the vagrant option is not really a good one because I have to switch hyper visor modes and reboot the machine every time to either use vagrant with virtual box or be able to use docker.

Seikilos commented 11 months ago

A little update after toying around with nomad and only be using raw_exec. I would prefer nomad over k8s but regarding linux containers on a windows machine I will probably stick to Rancher Desktop, which brings a working k3s out of the box and Rancher Dashboard comes along to provide a nice web ui.

I thought about trying nomad on a vps but since I can only can connect via public internet web interface to it (without building a vpn), I would be getting fully hit by "running nomad in production" with setting up mTLS and ACL, probably spending days figuring everything out. Then having issues like single node nomad server running into recovery mode, wiping all data and start again.

If I recall correctly, Armon Dadgar CTO of Hashicorp said, he regards nomad as a competitor to k8s (and I agree that k8s is complex and fragmented by various providers). I do not know, how big the "windows developer for linux containers" segment is (I would think it is largely growing since windows streamlines developing C#, etc. for linux containers), but as of now, I am one and nomad is unfortunately of no use for me. 😥

gthieleb commented 11 months ago

@Seikilos did you tested the solution of @das974 with using a seperate docker installation in WSL2 for the Linux Workload? . There is also the option to use podman [1] (or podman desktop) within WSL2 which also offers you running container workload with a (similar command line like docker) in Linux. For nomad a podman plugin exists [2].

[1] https://podman.io/docs/installation [2] https://github.com/hashicorp/nomad-driver-podman

Seikilos commented 11 months ago

@Seikilos did you tested the solution of @das974 with using a seperate docker installation in WSL2 for the Linux Workload? . There is also the option to use podman [1] (or podman desktop) within WSL2 which also offers you running container workload with a (similar command line like docker) in Linux. For nomad a podman plugin exists [2].

[1] https://podman.io/docs/installation [2] https://github.com/hashicorp/nomad-driver-podman

Hi, @gthieleb thanks for replying. I did and it did not work, but it was user error. Now the workaround of nomad being an agent in wsl seems to work. I explain what I did wrong to help others:

When I tried this, I had no docker driver in my WSL2 linux. But the issue was: The docker enabled WSL Linux is an ubuntu 20.04. I later installed a WSL2 Ubuntu 22.04 via app store for other things. The default setup was

Ubuntu 20.04 with docker support
Ubuntu 22.04 vanilla, no docker

So actually starting nomad client on 20.04 seems to work for now.

Another issue, we windows users are very likely to hit is: After rebooting our machine, the nomad server will go into recovery and lose all the data from the previous run. AFAIK this has something to do with its identity. The IP is part of it and since vEthernet WSL keeps assigning a new IP each time, I have to add a static IP to my windows host every time by calling

netsh interface ip add address "vEthernet (WSL)" 172.19.200.1 255.255.255.0

My client.hcl file then basically looks like

client {
  enabled = true
  servers = ["172.19.200.1:4647"]
}

I have yet to test, whether this works.

tgross commented 6 months ago

Hi folks! The Docker ecosystem has moved a little bit since this issue was originally opened, so I wanted to summarize the current state of the world and recommendations for running containers on Windows.

As of Docker 23.0, Docker has removed support for the original implementation for Linux containers on Windows (called "LCOW"). Nomad never really supported this well.
Running Linux containers on Docker Engine on Windows Server via WSL2 should work, assuming Nomad is running under WSL2 as well.
Running Linux containers on Docker Desktop on Windows should also work, with the caveats described in our FAQ for Docker Desktop
Running Windows containers on Windows Server with either Hyper-V or Process Isolation mode should work. We consider this to be the expected production environment for Nomad clients on Windows.

We're going to close this issue out as resolved. If you run into issues with current versions of Nomad and the specific scenarios described above, please feel free to open a new issue describing the symptoms you're seeing (and which scenario you're deploying in) and we'll try to get that sorted. Thanks everyone!

hashicorp / nomad