Consul support for Envoy on Windows

[joatmon08]

Feature Description

We’re looking for feedback on use cases and patterns for Consul support of Envoy on Windows. Envoy on Windows is now generally available.

Use Case(s)

If you would like to use Consul with Envoy on Windows, please :+1: and add a comment with your use case answering the following:

• Would you be running Consul servers on Windows?
• Would you be running Consul gateways on Windows (mesh, api gateway, or terminating)?
• Would applications be running on Windows VMs or containers?
• Would applications be running on Nomad or Kubernetes?
• What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.)
• Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)?

Any other information about the topology or architecture you’re looking to implement will help us determine how to best integrate Consul with Envoy on Windows.

Thank you!

[idrennanvmware]

High interest in this. We have a windows fleet that has no ability to run a service in the mesh.

Given we will be in this hybrid environment for a while, having the ability to run services on windows as part of mesh is desirable. Admittedly since it’s not been an option we have designed around it BUT if we were able to run a consul agent with mesh ingress gateway on our windows nodes it would be fantastic for our security story

Also we should call out that running containers on windows nodes is a sticky thing for us to navigate (but not impossible) and from what I can see only windows containers are supported (currently) by the envoy windows project

[brreisner]

• Would you be running Consul servers on Windows? Looking to run Consul on AKS with mixed node pools (windows and linux)
• Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? Mesh would be the current gap as we can run the ingress and terminating gateways on the linux nodes within the cluster, though feature parity would be ideal
• Would applications be running on Windows VMs or containers? Containers, AKS
• Would applications be running on Nomad or Kubernetes? Kubernetes AKS
• What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) AKS handles underlying Node OS (AKS uses Windows Server 2019 as the host OS version and only supports process isolation)
• Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Service mesh would be initial focus, again feature parity ideal

Ultimately we would like to implement a service mesh but until there is windows support it does not appear to make sense, as we would not treat all of our environment the same and it would become confusing as to what was handled how. Hoping consul can close this gap with a minimum viable product (service mesh support) for windows.

[faridshenassa2]

Hi,

Our use case is to use consul to create a layer 7 network / routing / control for both legacy windows/linux VMs as well as containers running under nomad. So basically simplify the traditional network/firewall/router/security by putting in an envoy in each vm then using consul to provision all VMs and containers to route through envoy with consuls as the control plane, and use consul ingress/egress gateway so only firewall rules in/out to internet as well as between sites is to / from envoy.

[mr-miles]

Also very interested in this. In future all the services will run in containers probably on linux/EKS/fargate but currently we have windows servers with services hosted in iis or running as full windows services. If we could connect this lot into the service mesh then it would make migrating to the promised land much simpler!

We have consul servers and ingress/egress gateways running in linux so it is the mesh part that is of most interest.

[Amier3]

@mr-miles Thanks for bringing this back to the forefront. I did some asking internally and we're still doing our investigations around this as apart of a larger initiative for more robust windows support.

It'd be super helpful to us if you ( and @kirooshu since I saw your upvote ) could take a second to answer the questions at the top of this post. We want to ensure we're building the right thing for y'all 😅

[faridshenassa2]

Hi All,

For our use case, we are working on the normal Consul/Nomad/Vault for container orchestration and service mesh as our next gen workload platfrom at mutliple private cloud/on-premise sites that are interconnected. with for now the non-enterprise edition so no federation and each site is its own independent cluster.

We then have existing legacy windows VMs doing work that we would like to connect to and include in the service mesh for Layer 7 routing, as well as interconnection to the mesh workloads and control access, routing, discovery from Nomad/Consul

Would you be running Consul servers on Windows? the consul controllers would be linux. we would run consul as an agent on windows VMs along with the envoy proxy to make that VM look like an entry in the mesh

Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? primarily running consul gateways for mesh, not planning for the windows VMs to be considered ingress or egress unless required. Would applications be running on Windows VMs or containers? VM Would applications be running on Nomad or Kubernetes? Nomad What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) 2k12 and above.
Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes, primarily for integration into ServiceNow, Sumologic.

Any other information about the topology or architecture you’re looking to implement will help us determine how to best integrate Consul with Envoy on Windows. (see above).

And thank you all for looking into supporting this model.

[idrennanvmware]

The above describes a very similar set of criteria to our platform too

[mr-miles]

Hi,

To summarise, our use case is to get services that are currently tied to windows hosts participating in the service mesh with low effort/no code changes, so we can then migrate them slowly to containers on linux which is our long term solution.

Would you be running Consul servers on Windows? No - the consul servers would be linux. We would run consul as an agent on windows VMs along with the envoy proxy to make that VM look like an entry in the mesh

Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? No, we run those on linux. Would applications be running on Windows VMs or containers? VM Would applications be running on Nomad or Kubernetes? No What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) 2k19 and above. Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes, we are already using these capabilities but not the service mesh.

Thanks for your help!

On Mon, Feb 7, 2022 at 10:19 PM Amier Chery @.***> wrote:

@mr-miles https://github.com/mr-miles Thanks for bringing this back to the forefront. I did some asking internally and we're still doing our investigations around this as apart of a larger initiative for more robust windows support.

It'd be super helpful to us if you ( and @kirooshu https://github.com/kirooshu )could take a second to answer the questions at the top of this post. We want to ensure we're building the right thing for y'all 😅

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/consul/issues/10286#issuecomment-1031993016, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQD4GEUMOJCU3NTKS5IOTU2BAPXANCNFSM45NPTMAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[firerain-fd]

Would you be running Consul servers on Windows? No Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? Mesh Would applications be running on Windows VMs or containers? VM or Bare metal Would applications be running on Nomad or Kubernetes? Perhaps in the future. What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) Windows Server 2019 and higher. Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes, we are now using Consul features, such as service discovery, health checking.

Our infrastructure consists of bare metal servers and virtual machines on Windows Server. On the servers, the services run in IIS or run as Windows Services. Each host has a Consul Agent which provides us with service discovery and health checking capabilities. I would like to establish communication between the services through a consul mesh (Envoy).

I would like to have one Envoy process running on each host for the mesh, with which services on one host can communicate with services on other hosts.

If the scheme above is difficult to implement, you can run for each service, a different instance of the Envoy process. For the sake of simplicity, this is a two-host scheme.

[mr-miles]

That second diagram (one envoy process per service) is the setup we are going for, since it makes migrating the actual services around and treating them completely independently much easier.

On Fri, Feb 18, 2022 at 2:25 PM firerain-fd @.***> wrote:

Would you be running Consul servers on Windows? No Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? Mesh Would applications be running on Windows VMs or containers? VM or Bare metal Would applications be running on Nomad or Kubernetes? Perhaps in the future. What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) Windows Server 2019 and higher. Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes, we are now using Consul features, such as service discovery, health checking.

Our infrastructure consists of bare metal servers and virtual machines on Windows Server. On the servers, the services run in IIS or run as Windows Services. Each host has a Consul Agent which provides us with service discovery and health checking capabilities. I would like to establish communication between the services through a consul mesh (Envoy).

I would like to have one Envoy process running on each host for the mesh, with which services on one host can communicate with services on other hosts. [image: image] https://user-images.githubusercontent.com/29374803/154698152-9865cfa2-c695-4fd9-9de3-aea70640bb72.png

If the scheme above is difficult to implement, you can run for each service, a different instance of the Envoy process. For the sake of simplicity, this is a two-host scheme. [image: image] https://user-images.githubusercontent.com/29374803/154700534-9a3c6715-eb5d-4af3-8f47-bc33d61c2c54.png

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/consul/issues/10286#issuecomment-1044608302, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQD4HOCM5UWJNZCWDX3KLU3ZJF7ANCNFSM45NPTMAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[danclien]

Would it be possible to allow consul connect envoy -sidecar-for=service to start the Envoy process on Windows? We're currently able to start the Envoy sidecar process manually by:

1. Using -bootstrap to generate the Envoy config
2. Making sure the config is encoded in UTF-8
3. Updating access_log_path to a Windows-friendly path
4. Starting envoy.exe manually by passing in the config generated

• Would you be running Consul servers on Windows?

No

• Would you be running Consul gateways on Windows (mesh, ingress, or terminating)?

No

• Would applications be running on Windows VMs or containers?

Both

• Would applications be running on Nomad or Kubernetes?

No

• What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.)

Windows Server 2019 and Windows Server 2022

• Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)?

Yes

Any other information about the topology or architecture you’re looking to implement will help us determine how to best integrate Consul with Envoy on Windows.

We're migrating away from Windows, but we want our older .NET Framework services to join the service mesh if possible. We're currently using HCP Consul with the Envoy ingress instances running on AWS ECS.

[baxor]

Use Case(s) If you would like to use Consul with Envoy on Windows, please 👍 and add a comment with your use case answering the following:

Would you be running Consul servers on Windows? No

Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? No

Would applications be running on Windows VMs or containers? Yes, docker containers

Would applications be running on Nomad or Kubernetes? Nomad

What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) Windows Server 2019 (via AWS)

Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes

[guifran001]

Use Case(s) If you would like to use Consul with Envoy on Windows, please 👍 and add a comment with your use case answering the following:

Would you be running Consul servers on Windows? YES

Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? Maybe. For now, containerized applications are forbidden in our environment. Since, there are no obvious way to have a private network between a service and its side car (other than having a VM by service which is not our case). One day, container will be accepted.

Would applications be running on Windows VMs or containers? VMs

Would applications be running on Nomad or Kubernetes? Nomad

What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) Windows Server 2019, 2022

Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? YES

[seabopo]

Use Case(s)

Would you be running Consul servers on Windows? Preferably Yes. We are a 99% Windows shop.

Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? Mesh is our primary use case. ingress and terminating are both future interests.

Would applications be running on Windows VMs or containers? Windows Containers on Windows Azure Docker Hosts.

Would applications be running on Nomad or Kubernetes? Nomad

What Windows server versions would you be interested in? (This will also depend on Envoy’s support of Windows server.) 1809, 2019 and 2022

Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes

[solvingj]

Would you be running Consul servers on Window? no Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? terminating Would applications be running on Windows VMs or containers? VMs Would applications be running on Nomad or Kubernetes? Kubernetes What Windows server versions would you be interested in? 2022+ Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? No

[maicalal]

Hi,

What is the latest on this ? do we have it in the main stream ? Was going through the tutorial here <https://developer.hashicorp.com/consul/tutorials/kubernetes/kubernetes-windows-nodes> It says : Note The Consul Kubernetes on Windows nodes feature is a preview release meant for testing purposes. This release does not currently support Consul ACLs or TLS.

Is it still in preview ?

[david-yu]

Hi folks, we are actively still developing Windows service mesh and have released the following:

• Windows Service Mesh beta for Consul on VMs (beta): https://developer.hashicorp.com/consul/tutorials/developer-mesh/consul-windows-workloads
• Consul on Kubernetes for Windows Workloads (alpha): https://developer.hashicorp.com/consul/tutorials/kubernetes/kubernetes-windows-nodes

If you have feedback on any of the two preview releases please post them here. We are hoping to get to GA state later this year with support for production environments with ACLs and TLS.

[maicalal]

For now what can be the workaround for the use case of mix workloads ?

Would you be running Consul servers on Window? no Would you be running Consul gateways on Windows (mesh, ingress, or terminating)? mesh Would applications be running on Windows VMs or containers? containers Would applications be running on Nomad or Kubernetes? Kubernetes What Windows server versions would you be interested in? 2022+ Would you be using Consul capabilities outside of service mesh (i.e., service discovery, health checking, etc.)? Yes

We can't go with alpha for prod setups hence trying for other workarounds.

I am trying to get some guidelines on how to go about having Consul agent and Envoy side-car containers on a windows pod manually and have it part of the service mesh. We have one service on windows and rest all are on linux base. We are not going with ACL for now.

Please advice.

[sevensolutions]

Iam working on an alternative IIS Plugin for nomad to run web applications in IIS on windows nodes. Beeing able to integrate them with the Service Mesh would be really great.

[vladbeha109]

Hi All,

We're looking for a service mesh which can span both VMs based on Windows OS(2022 Server) and Linux/Windows containers that we run in the K8s cluster. We see that Consul is still in preview mode, can you please provide any update regarding this matter?

From the information available on the internet it seems Envoy going to stop supporting Windows.

Regards,

[sevensolutions]

@vladbeha109 yes they are, unfortunately... :( https://www.envoyproxy.io/docs/envoy/latest/faq/windows/win_requirements