Cloud developer flow for Windows builds

[htuch]

In order to debug Windows builds (e.g. with CI failures), it's helpful to allow arbitrary Envoy developers the ability to perform Windows builds. I think there are two options here:

1. We have some pool of Windows VMs that we can share access to which are ready to go for Windows debugging.
2. We document, e.g. in README.md in https://github.com/envoyproxy/envoy/tree/master/bazel, how to setup a Windows build from scratch on AWS/GCP. This documentation should be detailed enough that following it will result in a valid Windows build environment, even for non-experienced Windows devs.

I think (2) is most likely to be viable, but (1) would be awesome if there was some way to achieve it.

CC @wrowe @sunjayBhatia @stedsome

[sunjayBhatia]

I was hoping something like the existing Linux dev container workflow would work, but looks like VSCode doesn't yet support windows containers: https://code.visualstudio.com/docs/remote/containers#_known-limitations

Setting up a remote docker host and having developers connect to that could be an option as well: https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost

[sunjayBhatia]

I can't see anywhere yet that Github codespaces supports windows, but that may be a future option

[wrowe]

We could potentially set this up with Azure, but the risk is that many tests including the ginormous mock tests still fail to compile in the small-ish azure containers. If this were restricted to only senior maintainers, it might be possible to rely on the gcp rbe environment for the heavy lifting. Knowing how large the resulting symbol tables are, I'm not sure if the Azure instances we've stood up for CI in the past are really sufficient. Agreed with your goals as stated.

[wrowe]

cc @envoyproxy/windows-dev to bring you all into this dialog

[sunjayBhatia]

Spent some time on this and it seems option (1) above is achievable with relatively low investment/moving parts. A first iteration would be a pool of Windows VMs maintainers can SSH to for rudimentary debugging.

• Works with the existing setup we have in ci-infra whereas VMs (currently for CI workers) are registered and deregistered
  • Maintain a pool of at least 1 claimable Windows workstation in AWS ASG
  • Maintainers get SSH key and can SSH to workstation, on SSH connection automate the VM detaching itself from ASG, ASG starts another standby VM available to claim
  • Likely the flow would be to clone the Envoy codebase, start up the dev container interactively (./ci/run_envoy_docker_windows.sh bash), use VSCode remote editing over ssh (or something similar) to edit files on the remote workstation, and iterate to debug
  • When user is done with the VM, terminate VM to clean it up
• Should be pretty much 1 VM running large majority of the time, relatively inexpensive
• Cloud providers often provide Windows VM images with the containers feature and Docker pre-installed, regardless, Docker is easy to install
• Only other dependencies we need to install are our lifecycle scripts, Git/bash, enable SSH/public key authentication

This first pass would enable maintainers to spin up a Docker container with the required tools to build on Windows. If RBE credentials are available, they will be able to take advantage of the remote cache/execution for builds. For debugging, initially this will enable simple log/trace debugging and help people fix compilation issues. We can work on attaching a remote debugger/debug server/hacking VSCode to provide remote devcontainer support (not currently supported on Windows) in the future if needed.

Links:

• https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse
• https://www.concurrency.com/blog/may-2019/key-based-authentication-for-openssh-on-windows
• https://github.com/envoyproxy/ci-infra#self-hosted-azp-agents
• http://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config.5?query=sshd_config#ForceCommand

[sunjayBhatia]

https://github.com/envoyproxy/envoy/pull/13282 is a direct example where we could have used this, should bump the priority of getting this done

[davinci26]

I have an unfortunate update. I have been looking some solutions internally at Microsoft and I looked into Codespaces for Windows. Unfortunately their timelines are not matching ours and this solution is not going to solve our problem for the foreseeable future.

[davinci26]

Another angle to this problem that @yanavlasov brought up during the community call on 14th of April is that we need Windows Server Images to debug the build tools and the CI. This is because docker works differently on Windows client and Windows Server.

The solution that we come up with should also address this problem

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.