Support Linux images in Docker for Windows

[schmichael]

Nomad only supports Docker for Windows 17.03.1 in Windows Container mode. Attempting to run Linux containers will fail because Nomad will create Windows-style volume bind paths instead of Linux-style.

We should:

• [ ] Detect Docker for Windows mode in the fingerprinter
  • [x] Requires updating go-dockerclient to support the undocumented Isolation key in the /info API - fsouza/go-dockerclient#649
  • Isolation: "hyperv" means Windows mode
  • Isolation: "" means Linux mode
  • [x] Add new attr.driver.docker.os_type to expose Docker mode to constraints
• [x] Create volume paths based on the image's OS
  • [x] Requires updating go-dockerclient to support the 1.29 InspectImage API - fsouza/go-dockerclient#649
• [ ] Adjust signal handling to be based on the runtime os, rather than the nomad-client os.

Stretch goal:

• [ ] Vary the output of nomad init based on Docker for Windows mode (redis:3.2 for Linux, redis:nanoserver for Windows)

[KZachariassen]

Any news on this one, we could really use this!

[schmichael]

@krjensen I'm afraid we haven't prioritized it on our roadmap yet since a Windows node running Docker in Linux mode doesn't offer any functionality not already provided by running Docker on a Linux node. We were curious if there were enough people running in homogenous Windows environments to make Linux mode on Windows compelling.

[KZachariassen]

I totally understand that.

So why do I need this: We are a .net developer house and has just begun to look at .net core that will run on Linux.

So every developer use windows 10 with docker, to build .net core applikations. We are amning to use nomad in produktion and there for the need to test it local, so a developer can write and test the .nodmad file. Many services will contain multiple containers, so they need to compose. I would prefer the need for only a nomad definition, that works both local and in production and not a situation where I need both a docker compose and a nomad definition.

[frundh]

We are also a Windows-shop facing the same issue when it comes to local development and testing, so this feature would be a nice addition.

Since this is only required for local development, I have experimented with some alternative solutions/hacks that I can share if anybody is interested:

Docker Image https://github.com/frundh/nomad-mobyvm

Nomad.exe https://github.com/frundh/nomad/releases/tag/v0.6.2.1-moby

[thasmo]

Would love to see this supported too, although I don't use it in a business-context.

[bartuer]

It will became pretty common to run Linux docker image on Windows, we seldom run Windows image on Windows inside Microsoft.

[automaticgiant]

just for tracking @schmichael, https://github.com/fsouza/go-dockerclient/issues/649 is done so the remaining work is in-repo, and confined to the listed plan now, right?

[adamsmithkld]

This is even more important now that Docker for Windows supports side-by-side Linux and Windows containers (via the experimental feature). We are also a Windows development shop that builds Linux Docker containers. The only thing missing is Nomad support for Linux containers on Docker for Windows.

[renevo]

Is there any traction on this, currently even with drive sharing in windows, nomad is unable to allocate a job due to permission denied on mounting drives.

2018-08-08T21:12:09.945-0700 [DEBUG] plugin: plugin process exited: path=D:\tools\nomad.exe
    2018/08/08 21:12:09.945889 [WARN] client: failed to start task "redis" for alloc "6354d373-9fd5-c0f3-a96c-09bf0a737f6f": Failed to create container: API error (500): Mount denied:
The source path "D:/nomad.d/nomad/alloc/6354d373-9fd5-c0f3-a96c-09bf0a737f6f/alloc:c"

I am able to mount volumes when running containers natively with docker for windows (linux containers, non-experimental), but nomad isn't able to mount.

PS: I am running nomad on windows, instructing to use the pipefile.

data_dir = "d:\\nomad.d\\nomad"
datacenter = "dc1"

client {
    enabled = true
    network_interface = "Ethernet"

    options {
        "docker.endpoint" = "npipe:////./pipe/docker_engine"
    }
}

server {
    enabled = true

    bootstrap_expect = 1
    data_dir = "d:\\nomad.d\\nomad\\server"
}

I get the error:

failed to start task "redis" for alloc "a7feb856-514e-9d52-fb2b-ca87f55172d5": Failed to create container: API error (500): Mount denied: The source path "D:/nomad.d/nomad/alloc/a7feb856-514e-9d52-fb2b-ca87f55172d5/alloc:c" doesn't exist and is not known to Docker

However, the directory is definately available:

[jzupka]

exactly same to me

[sujeetsm]

upvote this one .. windows now supports homogeneous support for linux/windows containers.

[gblok]

How to make work nomad with Linux images in Docker for Windows? It is already possible?

[cgbaker]

We don't have any immediate plans to support this, but we are tracking it in our backlog.

[atrauzzi]

Chiming in to say that I've just encountered this issue. A bit disappointed after reading the last comment.

Not being able to run linux containers via nomad on Windows completely blocks what I'm currently working on. Worse still is that the first container I tried to run doesn't even have bind mounts configured!

Microsoft and Docker have been putting a lot of work into ensuring that Docker Desktop gets as much out of the way as possible. Even so far as allowing Windows and linux containers to run at the same time. It doesn't feel like too much to suggest that nomad add support for this scenario. Even if only to remain consistent with the direction of the ecosystem.

[schmichael]

Even so far as allowing Windows and linux containers to run at the same time.

That's fantastic and was not the case when this ticket was originally written! Originally Docker on Windows was modal and only supported one image type at a time. That may nullify the Add new attr.driver.docker.container_mode to expose Docker mode to constraints checkbox above and make implementation even easier.

[nbpalomino]

I'm also waiting for implementing this feature... currently trying to use Nomad with Docker for Windows on my Windows 10 box...

[ilyaiqoqo]

I'm also waiting I am waiting for this feature. we are running batch jobs that should be agnostic to where they run

[amfern]

I am able to run Linux container when the daemon is in windows containers mode, by enabling experimental features in docker edge.

I think nomad should also support this and not set the driver to an unhealthy state

[ilyaiqoqo]

I am attempting to make lcow work, I've extracted some patches i made into three PRs, would appreciate feedback

• [x] fixes invalid network mode when running linux container on windows #5999
• [x] allows specifying relative binding with backslashes or forward slashes #6000
• [x] update destination regex to allow unix destination path in container https://github.com/hashicorp/nomad/pull/6001

[karlem]

Is there any update on this issue please?

We're planning to use Nomad for very interesting use case but support of Linux containers on Windows is must.

Thanks

[tgross]

Hi @karlem! The patches we received from @ilyaiqoqo should be enough to allow support, but I admit it's not an area we have well-tested yet. I recently added Windows 2016 (Datacenter) to our end-to-end test suite, but that supports only Windows containers on Windows. We've got upcoming work planned to add Windows 2019 targets to e2e testing that will make sure we've got well-tested support there as well.

(I have some open issues tracking that work on a private GitHub board that I'll move over into this repo shortly so that you can keep an eye on them.)

[karlem]

Hi @tgross! Thank you very much for your response. That sounds pretty good. I will keep an eye on the issues.

Have a good day!

[atrauzzi]

@tgross - Is that using Docker for Windows (the desktop app)?

[tgross]

No, it's the engine for the server. My understanding is that the underlying virtualization problems to solve are the same so support for Windows 2019 production environments should solve for those desktop use cases as well. But the desktop app isn't likely to be something we'll fully support with testing from a practical standpoint.

[atrauzzi]

Hmm. I had been originally hoping this would be supported on desktop for local development.

[karlem]

Yeah I didn't manage to run Linux container on Windows 10 with Hyper-V at all.

Just got error message: Docker is configured with Linux containers; only Windows containers are supported

@tgross is there potentially a way we could contribute to make it happen??? Since my company is planning to use Nomad as a base for grid compute solution we would need to support Windows 10 as well.

Thanks

[tgross]

Looks like that error message is bubbling up from fingerprint.go#L168-L178 which was added in https://github.com/hashicorp/nomad/pull/5356 -- before the patches we merged from @ilyaiqoqo.

We'll need to remove that check (at the very least), before Linux containers on Windows can be supported. It looks like we'll still need to have some fix for signal handling as well. I've moved this issue into a better spot in internal tracking to make sure we put some eyes on this. And @karlem if your team is interested in contributing patches for the open problems in this ticket we'd be happy to accept them!

[karlem]

Ok perfect.. I have spoken to team members and we're more than happy to help. We will have a look at it probably tomorrow. We will try to remove the fingerprint, schedule some Linux containers and see what happens.

Thanks @tgross for being responsive! Have a good day

[Ankurkh1]

Any update on this please. Because of this issue we are considering moving to docker swarm. I just love the Hashicorp stack. Badly want to stay with it.

[jlarriba]

I have been working on this issue a little bit and I managed to get Nomad to run a linux container in Windows Server 2019 correctly, in a very embrionary state. It does not accept extra volumes, I only solved the first step of the alloc, local and secrets mandatory volumes. But with my changes, the Windows containers stop working.

In current Windows Server 2019, Docker runs Linux images and Windows images on a totally transparent way to fsouza/go-dockerclient. Currently, in the fingerprint, there is this check:

if runtime.GOOS == "windows" && dockerInfo.OSType == "linux"

which always fails in Windows Server 2019, as Docker for Windows reports OSType as "windows" even when running linux images. This means that the Docker is configured with Linux containers; only Windows containers are supported message is never shown on Server 2019 and we go directly to the volume mount errors.

I have reviewed back and forth the identification capabilities of fsouza/go-dockerclient, but I have not been able to discover a way to identify when a Windows server is running a Linux image, all the parameters are identical than the ones shown when running a Windows image.

Any ideas?

[jlarriba]

OK, I found it. Inspecting a container gives us the "Platform" parameter, which is "linux" or "windows", depending on the type of image we are using.

[arieradle]

OK, I found it. Inspecting a container gives us the "Platform" parameter, which is "linux" or "windows", depending on the type of image we are using.

Does this mean that there will be some basic solution for Linux containers on windows soon? I think if you can roll out a very basic solution, even as beta, this would be a major step forward for Nomad.

[tgross]

@arieradle #7850 was just merged to master, so that'll go out in the next patch release 0.11.2 (I can't give you an exact date on that other than "soon"). I'd like to add some end-to-end testing of that patch in our nightly build as well. If you'd like, feel free to build the latest from master and give it a spin yourself!

[arieradle]

Kudos. In any case, IMHO the documentation should be amended to emphasize current shortcoming, and the fact that it is about to be fixed. It's not really an intuitive issue.

Will this fix work on Win 10 (Hyper-V isolation)? If so, beginning at which release? 1703/1803/1903/1909.

If there are more details regarding the Docker driver compatibility issues with Docker/Host OS/Container platform - please add them to documentation.

[jlarriba]

Will this fix work on Win 10 (Hyper-V isolation)? If so, beginning at which release? 1703/1803/1903/1909.

@arieradle Win10 was never a target of this feature, as the main way to run Linux containers is via the Hyper-V Linux VM ("Switch to Linux Containers"). However, motivated by the curiosity, I tried to schedule Linux containers in Windows 10 using Nomad and they ran flawlessly:

┌[jlarriba@WINLAPTOP]
└[C:]> docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS
               NAMES
c757b35531d5        nginx               "nginx -g 'daemon of…"   12 minutes ago      Up 11 minutes       0.0.0.0:8080->80/tcp, 0.0.0.0:8080->80/udp   nginx-0a5255c2-e925-5988-1d8d-d27a5995232f

My Win10 version:

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      18363  0

[BasixKOR]

I think this is pretty crucial for developers using Windows to actually try Nomad. I'm currently launching client agent on WSL 2 as a workaround, and it is really a hassle to set up all of this.

[ansario]

@jlarriba how did you accomplish that?

[jlarriba]

@BasixKOR @ansario sorry, but I do not understand what you want.

A PR that fixes this issue was merged to master on May 4th: Run Linux Images (LCOW) and Windows Containers side by side and it was included from Nomad version 0.12 onwards. So any current version should be able to run both LCOW and Windows Containers side-by-side.

[ansario]

@jlarriba is it unsupported on Windows 10? You mentioned it worked flawlessly and I wanted to know how you got it to work. Right now the latest version of Nomad says to switch to Windows containers.

[jlarriba]

@ansario I am not the one who says what it is officially supported or not on Nomad, as I am not a Hashicorp employee. According to my tests (see my previous comment on the subject), the changes I implemented allows Nomad to schedule Windows Containers on Windows 10.

However, I still do not know what your intention is and what exactly do you expect from Nomad in Windows 10, so it is complicated to provide more help.

[tgross]

Officially we want to support Windows 2019; my understanding is that it should work but we haven't invested time into making certain of that. That's actually something I'm going to be working on over this week and next. Windows 10 isn't a production environment, so if we can get to a point where we're sure of Windows 2019 LCOW, we're likely to say "yes this should work on Windows 10 but it's considered a development environment," similar to our macOS support.

[jlarriba]

@tgross if you think I can help, please feel free to ping me.

[ansario]

Perfect, thanks for the response @tgross and @jlarriba, let me know if I can help with testing.

[avishnyakov]

This ticket summarises why I now have dual boot with Ubuntu on my Windows 10 laptop. At the end, it was worth it, less challenges with docker and other utilities is always good.

But still, would be nice to have Nomad supporting Docker Desktop with Linux containers on Windows. As mentioned, mostly dotnet core development scenarios and overall Nomad playground.

[BogusDude99]

Can we please have an update for this issue?

[tgross]

Hi @BogusDude99. Generally speaking if we have an update on the status of an issue, you'll see it in the issue. The work I referred to in https://github.com/hashicorp/nomad/issues/2633#issuecomment-717900145 got done but it's still not fully-tested E2E.

[lutetium174]

Hey @tgross

Is there a preview version that we can test and provide feedback on in the meantime?

[tgross]

@lutetium174 the work described there was for testing, as the primary issue at this point is whether we're certain that LCOW is supported (mostly because testing LCOW means spinning up bare metal machines with Windows, which isn't part of our usual testing environment). You can already give Nomad 1.0.4 a spin today and see if LCOW is working for your environment.

[lutetium174]

@tgross Thanks for the suggestion but it seems, at least for Windows 10 using Linux containers WSL2, unfortunately not.

I'll continue to monitor this thread and issue.

[jlarriba]

@lutetium174 The feature target was not WSL2, but LCOW, a very promising feature included in Windows Server 2019 that allowed to run Windows Containers natively along with Linux Containers via the Hyper-V Containers feature. The API of LCOW is very similar to the one used by Docker in Windows, so supporting it took very little effort. Even if it was not the target, I checked (manually, this was never officialy tested as @tgross said) that it also made Nomad work with the Linux Containers feature provided by the Docker Desktop Windows 10 application.

Even if last year LCOW looked like a promising technology and the "way-to-go" for Docker on Windows support, it never left experimental phase, and it seems to look more and more abandoned. Since then, the Docker team focused their efforts for Linux Containers on WSL2. See this issue.

WSL2 however, is a completely different technology that has nothing to do with LCOW, so it would require additional effort to add support to that in Nomad.