Building Agon on Windows

[markmandel]

Out of the box, Agon will not build with Windows.

I expect best effort will be to force people to use WSL to do this (which I don't think is too onerous)

Even with WSL, I expect that some of these will break, especially the minikube development workflow.

This is a high level ticket for tracking these issues, and resolving them.

Docker

May need to create a $DOCKER env var that switches to docker.exe on windows Should be relatively trivial to inspect the OS and switch out as needed.

Minikube

Best option: After creating the cluster with make minikube-test-cluster, run eval $(minikube docker-env) and work with minikube as per described. We've set it up so the build image gets transferred in on minikube start, and since the VM knows it's own external IP, everything should work at this point as expected.

[cyriltovena]

Hi Mark,

I've actually spent some time on this already. And it was not successful, I used wsl and docker for windows connected to hyper-v. My WSL has the docker cli using the windows docker deamon via tcp and it cannot mount linux path, it expects only windows path. Another option that might work but I found a bit too bad would be to not use the docker for windows (hyper-v) and use a the virtualbox(might not also work). In my opinion this goes against the tide, I'm used to docker for windows and losing it makes me feel bad (I love it so far and it's the de-facto in the docker community, docker toolbox is way more painful to use)

Unfortunately from my research it will never work if we don't change a bit how the build works, now I don't really know which direction we want to go for, a second build system for windows or a common for every platform ? I'm sure both are possible.

Let me know and I can take it from there.

Cyril.

PS: Also do note that the same issue will happens with CI, usually CI uses docker dind and mounting in dind is clunky since you're mounting the host not the container host, not sure if I'm clear ? but that's another subject ;).

[markmandel]

Thanks for doing work on this @Kuqd !

So I'm using this build system now for CI with dind, and it's fine (at least on Container Builder - works with very little trouble. Have a look at cloudbuild.yaml for reference.

Ideally, I'd like to have a single build system that works across the board (switch out some pieces based on what OS people are on) - but also agree, I don't want Windows users to have to install virtualbox just to be able to work on this, that would incur a great deal of friction.

Can you share the logs of what happens when you run make build for example inside WSL?

Curious also what adjustments you made to the Makefile to get even part of the way there?

[cyriltovena]

The steps I've made :

1 - Install docker for windows (I already had that) and allow localhost deamon access via the settings 2 - Install WSL and install docker cli (https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/) 3 - install make in WSL 4 - set the docker cli in WSL to use the windows deamon export DOCKER_HOST=localhost uses docker info to make sure it works.

The build of the build image worked then the logs are showing this :

root@MTL-BH744:/mnt/f/src/github.com/agonio/agon/build# make
mkdir -p /mnt/f/src/github.com/agonio/agon/build//.kube
mkdir -p /mnt/f/src/github.com/agonio/agon/build//.config/gcloud
docker run --rm -e "CGO_ENABLED=0" -v /mnt/f/src/github.com/agonio/agon/build//.config/gcloud:/root/.config/gcloud -v /mnt/f/src/github.com/agonio/agon/build//.kube:/root/.kube -v /mnt/f/src/github.com/agonio/agon:/go/src/github.com/agonio/agon --entrypoint=go agon-build:76f0b220a6 build \
                -o /go/src/github.com/agonio/agon/gameservers/controller/bin/controller -a -ldflags "-X github.com/agonio/agon/pkg.Version=0.1-1601c83" -installsuffix cgo github.com/agonio/agon/gameservers/controller
can't load package: package github.com/agonio/agon/gameservers/controller: cannot find package "github.com/agonio/agon/gameservers/controller" in any of:
        /usr/local/go/src/github.com/agonio/agon/gameservers/controller (from $GOROOT)
        /go/src/github.com/agonio/agon/gameservers/controller (from $GOPATH)
make: *** [build-gameservers-controller-binary] Error 1

You get the idea, nothing has been mounted, and nothing will ever be mounted with a linux path, I've dig a lot around this.

I also have a tutorial to get minikube working on hyper-v in a private repo, I'll see if I can make it public for you or may be you want it in a wiki page somewhere here. It's important for windows developer to keep hyper-v because you can't have both hyper-v and virtualbox, and for me it will always be hyper-v since it the best docker experience you can have.

Cyril.

[markmandel]

Oh that's good to know - I'm working on #30 right now - both to have Agon work on Minikube, and also have it as a development platform. Knowing we need to stay on hyper-v is very useful.

Thanks for digging into this - you may have seen this, but this approach looks promising: https://github.com/Microsoft/WSL/issues/1854

I'm downloading a Windows 10 vm now (yes, that will be Linux->Windows->WSL->Docker :wink:), this may be a weekend hack project, just to see what breaks/changes need to be made.

I was just doing some quick reading on running minikube on Windows with hyper-v, and it didn't look like the best experience, so glad you have it worked out. That being said, Kubernetes support should be coming in Docker for Windows natively in the near future, so it may become a moot point (but who knows when that will actually show up)

[cyriltovena]

I'll test to bindmount and symlink to see if it helps.

I don't think you can get hyper-v working in a vm but you will tell me soon ;)

[markmandel]

Here's a whole setup script - https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly

It's looking like it's a fairly common practice, from my super limited research. (I wonder if we should build it into the Makefile operations for windows, if it doesn't exist, create it)

[cyriltovena]

Yes this is exactly what I did, symlink doesn't work because make use the real path in the end.

I'm trying to have the bind mount working, but it doesn't, might be related to my WSL version as I'm lock in windows update from Ubisoft. I know that Eric had the approval to get the fall creator update, I'll ask him.

EDIT: Yep I'm doomed bind mount doesn't work for me.

[cyriltovena]

I didn't wanted to changed your build, but something we can do is bake make in your build image and forget about WSL. It's one less dependency ..

Basically embrace docker and make it the only dependency to build. If it works only with docker it will work every where there is a docker host.

[markmandel]

So moving everything into Docker is an interesting idea - but we'd still need some kind of target system - I'm not sure if we can get away from Make.

With the build system, my thinking is that there is OS detection (such as a simpler version of this) , and it drops in a OS specific include that allows for overwrites to occur to make sure things work correctly on each OS. This feels like a simpler answer to me (at least, for now)

I believe if the Windows Makefile include did a simple find and replace on the prefix of agon_path and build_path for Windows to point to the correct symlink place, that would resolve at least the initial issue.

I was going to attempt this - but apparently you can't install Docker on Win in a VirtualBox VM :cry: , so I'm going to have to grab an actual laptop to install Windows on to take an initial stab at this.

[cyriltovena]

Ok so we have two options:

• find a more platform independent build target system (ninja, docker, docker-compose)
• attempt to rewrite /mnt/f to /f on windows only

the reason why this is happening is because you send the code via mount, usually to build docker images you use the build context to pass either the code (and then use a multi-stage build if there is a lot of tools required) or directly artefacts.

EDIT: I've managed to go further with the path rewrite. pushing a branch for better collaboration EDIT2: I'm stuck somewhere else, when sending the artifacts to deamon, I'm hitting exactly this https://github.com/docker/for-win/issues/576 and https://github.com/Microsoft/WSL/issues/1123 seems that the only way to fix is a windows update which I can't for now.

For those who have a windows up to date you can keep going using my branch

[markmandel]

So I'm running this now on WSL, and so far, both make test and make build are working perfectly.

I'm going to test out the rest of the the build and development process.

I wonder if it's because I'm /c ?

I followed this guide: https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly (Without docker-compose)

Here are my installed versions -

Ubuntu: 20170830.1 build of Ubuntu 16.04 LTS

Docker Version:

Client:
 Version:       17.12.0-ce
 API version:   1.32 (downgraded from 1.35)
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:11:19 2017
 OS/Arch:       linux/amd64

Server:
 Engine:
  Version:      17.09.1-ce
  API version:  1.32 (minimum version 1.12)
  Go version:   go1.8.3
  Git commit:   19e2cf6
  Built:        Thu Dec  7 22:28:28 2017
  OS/Arch:      linux/amd64
  Experimental: true

markmandel@DESKTOP-BDM5UCP:/c/Users/Mark/Documents/workspace/agon/build$ make build

mkdir -p /c/Users/Mark/Documents/workspace/agon/build//.kube mkdir -p /c/Users/Mark/Documents/workspace/agon/build//.config/gcloud docker run --rm -e "CGO_ENABLED=0" -v /c/Users/Mark/Documents/workspace/agon/build//.config/gcloud:/root/.config/gcloud -v ~/.kube:/root/.kube -v /c/Users/Mark/Documents/workspace/agon:/go/src/github.com/agonio/agon --entrypoint=go agon-build:6c2ef6cd74 build \ -o /go/src/github.com/agonio/agon/gameservers/controller/bin/controller -a -ldflags "-X github.com/agonio/agon/pkg.Version=0.1-f12945a" -installsuffix cgo github.com/agonio/agon/gameservers/controller docker build /c/Users/Mark/Documents/workspace/agon/gameservers/controller/ --tag=gcr.io/agon-images/gameservers-controller:0.1-f12945a Sending build context to Docker daemon 36.23MB Step 1/5 : FROM alpine:3.6 3.6: Pulling from library/alpine 1160f4abea84: Pull complete Digest: sha256:42c18571af45ffc440bd5127de9ad604f1683e4799dbf7bf588aaee8f95c18f1 Status: Downloaded newer image for alpine:3.6 ---> e2cd449cde75 Step 2/5 : COPY ./bin/controller /home/agon/controller ---> 2549d181922b Step 3/5 : RUN apk --update add ca-certificates && adduser -D agon && chown -R agon /home/agon && chmod o+x /home/agon/controller ---> Running in 6284706b51b6 fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz (1/1) Installing ca-certificates (20161130-r2) Executing busybox-1.26.2-r9.trigger Executing ca-certificates-20161130-r2.trigger OK: 5 MiB in 12 packages ---> 736e72e83826 Removing intermediate container 6284706b51b6 Step 4/5 : USER agon ---> Running in afea29606f7d ---> 0b935cd4398c Removing intermediate container afea29606f7d Step 5/5 : ENTRYPOINT /home/agon/controller ---> Running in d7a1111029da ---> b9a4548fc488 Removing intermediate container d7a1111029da Successfully built b9a4548fc488 Successfully tagged gcr.io/agon-images/gameservers-controller:0.1-f12945a docker run --rm -e "CGO_ENABLED=0" -v /c/Users/Mark/Documents/workspace/agon/build//.config/gcloud:/root/.config/gcloud -v ~/.kube:/root/.kube -v /c/Users/Mark/Documents/workspace/agon:/go/src/github.com/agonio/agon agon-build:6c2ef6cd74 go build \ -o /go/src/github.com/agonio/agon/gameservers/sidecar/bin/sidecar.linux.amd64 -a -ldflags "-X github.com/agonio/agon/pkg.Version=0.1-f12945a" -installsuffix cgo github.com/agonio/agon/gameservers/sidecar docker run --rm -e "GOOS=darwin" -e "GOARCH=amd64" -v /c/Users/Mark/Documents/workspace/agon/build//.config/gcloud:/root/.config/gcloud -v ~/.kube:/root/.kube -v /c/Users/Mark/Documents/workspace/agon:/go/src/github.com/agonio/agon agon-build:6c2ef6cd74 go build \ -o /go/src/github.com/agonio/agon/gameservers/sidecar/bin/sidecar.darwin.amd64 -ldflags "-X github.com/agonio/agon/pkg.Version=0.1-f12945a" github.com/agonio/agon/gameservers/sidecar docker run --rm -e "GOOS=windows" -e "GOARCH=amd64" -v /c/Users/Mark/Documents/workspace/agon/build//.config/gcloud:/root/.config/gcloud -v ~/.kube:/root/.kube -v /c/Users/Mark/Documents/workspace/agon:/go/src/github.com/agonio/agon agon-build:6c2ef6cd74 go build \ -o /go/src/github.com/agonio/agon/gameservers/sidecar/bin/sidecar.windows.amd64.exe -ldflags "-X github.com/agonio/agon/pkg.Version=0.1-f12945a" github.com/agonio/agon/gameservers/sidecar docker build /c/Users/Mark/Documents/workspace/agon/gameservers/sidecar/ --tag=gcr.io/agon-images/gameservers-sidecar:0.1-f12945a Sending build context to Docker daemon 100.9MB Step 1/5 : FROM alpine:3.6 ---> e2cd449cde75 Step 2/5 : COPY ./bin/sidecar.linux.amd64 /home/agon/sidecar ---> 6f18bcf9d603 Step 3/5 : RUN apk --update add ca-certificates && adduser -D agon && chown -R agon /home/agon && chmod o+x /home/agon/sidecar ---> Running in 78b9ff0cbf18 fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz (1/1) Installing ca-certificates (20161130-r2) Executing busybox-1.26.2-r9.trigger Executing ca-certificates-20161130-r2.trigger OK: 5 MiB in 12 packages ---> 190d8aee0529 Removing intermediate container 78b9ff0cbf18 Step 4/5 : USER agon ---> Running in 45b17572517b ---> 0c65571f5765 Removing intermediate container 45b17572517b Step 5/5 : ENTRYPOINT /home/agon/sidecar ---> Running in 4929f6c0a926 ---> efe976212b87 Removing intermediate container 4929f6c0a926 Successfully built efe976212b87 Successfully tagged gcr.io/agon-images/gameservers-sidecar:0.1-f12945a docker run --rm -v /c/Users/Mark/Documents/workspace/agon/build//.config/gcloud:/root/.config/gcloud -v ~/.kube:/root/.kube -v /c/Users/Mark/Documents/workspace/agon:/go/src/github.com/agonio/agon -w /go/src/github.com/agonio/agon/sdks/cpp --entrypoint make agon-build:6c2ef6cd74 build install archive VERSION=0.1-f12945a mkdir /go/src/github.com/agonio/agon/sdks/cpp//bin g++ -std=c++11 -I/usr/local/include -pthread -fPIC -c -o sdk.grpc.pb.o sdk.grpc.pb.cc g++ -std=c++11 -I/usr/local/include -pthread -fPIC -c -o sdk.pb.o sdk.pb.cc g++ -std=c++11 -I/usr/local/include -pthread -fPIC -c -o sdk.o sdk.cc g++ sdk.grpc.pb.o sdk.pb.o sdk.o -L/usr/local/lib -lgrpc++_unsecure -lgrpc -lprotobuf -lpthread -ldl -shared -o /go/src/github.com/agonio/agon/sdks/cpp//bin/libagonsdk.so ar rcs /go/src/github.com/agonio/agon/sdks/cpp//bin/libagonsdk.a sdk.grpc.pb.o sdk.pb.o sdk.o cp /go/src/github.com/agonio/agon/sdks/cpp//bin/libagonsdk.* /usr/local/lib mkdir -p /usr/local/include/agon cp /go/src/github.com/agonio/agon/sdks/cpp//*.h /usr/local/include/agon/ ldconfig rm /go/src/github.com/agonio/agon/sdks/cpp//bin/argonsdk-0.1-f12945a-dev-linux-arch_64.tar.gz rm: cannot remove '/go/src/github.com/agonio/agon/sdks/cpp//bin/argonsdk-0.1-f12945a-dev-linux-arch_64.tar.gz': No such file or directory make: [archive] Error 1 (ignored) Makefile:65: recipe for target 'archive' failed rm /go/src/github.com/agonio/agon/sdks/cpp//bin/argonsdk-0.1-f12945a-runtime-linux-arch_64.tar.gz rm: cannot remove '/go/src/github.com/agonio/agon/sdks/cpp//bin/argonsdk-0.1-f12945a-runtime-linux-arch_64.tar.gz': No such file or directory Makefile:65: recipe for target 'archive' failed cp /usr/local/lib/libgrpc.so.5 /go/src/github.com/agonio/agon/sdks/cpp//bin/ make: [archive] Error 1 (ignored) cp /usr/local/lib/libprotobuf.so.15 /go/src/github.com/agonio/agon/sdks/cpp//bin/ cp /usr/local/lib/libagonsdk.so /go/src/github.com/agonio/agon/sdks/cpp//bin/ cp /usr/local/lib/libgpr.so.5 /go/src/github.com/agonio/agon/sdks/cpp//bin/ cp /usr/local/lib/libgrpc_unsecure.so.5 /go/src/github.com/agonio/agon/sdks/cpp//bin/ cd /go/src/github.com/agonio/agon/sdks/cpp//bin && tar cvf argonsdk-0.1-f12945a-runtime-linux-arch_64.tar.gz * libagonsdk.a libagonsdk.so libgpr.so.5 libgrpc.so.5 libgrpc_unsecure.so.5 libprotobuf.so.15 cd /usr/local && tar cvf /go/src/github.com/agonio/agon/sdks/cpp//bin/argonsdk-0.1-f12945a-dev-linux-arch_64.tar.gz lib include lib/ lib/libagonsdk.so lib/libagonsdk.a lib/python2.7/ lib/python2.7/site-packages/ lib/python2.7/dist-packages/ lib/libgpr.a lib/libgrpc++.so.1 lib/libgrpc++_reflection.a lib/libgrpc++_cronet.so lib/libgrpc.a lib/libgrpc++_reflection.so.1.8.2 lib/libgrpc++_error_details.so.5 lib/libgrpc.so.5 lib/libgrpc++_unsecure.so.5 lib/libgrpc++_reflection.so.5 lib/libgrpc++_error_details.a lib/libgrpc_unsecure.so.5 lib/libgrpc++_cronet.so.5 lib/libgrpc.so.5.0.0 lib/libprotobuf-lite.so.15 lib/libgrpc++_unsecure.so.1.8.2 lib/libgrpc++.a lib/libgrpc++.so lib/libgpr.so.5.0.0 lib/libgrpc++_error_details.so lib/libgrpc++.so.5 lib/libgrpc_unsecure.so lib/libgrpc++_error_details.so.1 lib/libgrpc_unsecure.so.5.0.0 lib/libgrpc_cronet.so.5 lib/libgrpc++_reflection.so lib/libgrpc_cronet.so lib/libprotobuf.so.15 lib/libgrpc++_unsecure.so.1 lib/libgrpc++_cronet.so.1.8.2 lib/libprotobuf.a lib/libprotoc.so.15.0.0 lib/libgrpc++_cronet.a lib/libgrpc++_cronet.so.1 lib/libprotobuf-lite.a lib/libprotobuf-lite.so.15.0.0 lib/libprotoc.so.15 lib/libprotobuf.la lib/libgpr.so.5 lib/libgrpc++_unsecure.a lib/libgrpc_unsecure.a lib/libprotoc.la lib/libprotoc.so lib/libgrpc++_reflection.so.1 lib/libgpr.so lib/libgrpc_cronet.so.5.0.0 lib/libprotobuf-lite.so lib/libprotobuf.so lib/libgrpc++.so.1.8.2 lib/libprotobuf.so.15.0.0 lib/libgrpc++_unsecure.so lib/libprotobuf-lite.la lib/libprotoc.a lib/pkgconfig/ lib/pkgconfig/grpc.pc lib/pkgconfig/grpc++_unsecure.pc lib/pkgconfig/grpc++.pc lib/pkgconfig/grpc_unsecure.pc lib/pkgconfig/protobuf-lite.pc lib/pkgconfig/protobuf.pc lib/libgrpc.so lib/libgrpc_cronet.a lib/libgrpc++_error_details.so.1.8.2 include/ include/agon/ include/agon/sdk.grpc.pb.h include/agon/sdk.h include/agon/sdk.pb.h include/grpc/ include/grpc/slice.h include/grpc/status.h include/grpc/support/ include/grpc/support/atm_windows.h include/grpc/support/atm.h include/grpc/support/cmdline.h include/grpc/support/log.h include/grpc/support/tls_gcc.h include/grpc/support/port_platform.h include/grpc/support/time.h include/grpc/support/sync_windows.h include/grpc/support/sync.h include/grpc/support/atm_gcc_sync.h include/grpc/support/host_port.h include/grpc/support/tls_pthread.h include/grpc/support/sync_posix.h include/grpc/support/histogram.h include/grpc/support/tls.h include/grpc/support/thd.h include/grpc/support/cpu.h include/grpc/support/atm_gcc_atomic.h include/grpc/support/subprocess.h include/grpc/support/tls_msvc.h include/grpc/support/sync_generic.h include/grpc/support/workaround_list.h include/grpc/support/useful.h include/grpc/support/alloc.h include/grpc/support/avl.h include/grpc/support/sync_custom.h include/grpc/support/string_util.h include/grpc/support/log_windows.h include/grpc/slice_buffer.h include/grpc/grpc_security_constants.h include/grpc/grpc_cronet.h include/grpc/fork.h include/grpc/byte_buffer_reader.h include/grpc/byte_buffer.h include/grpc/compression.h include/grpc/grpc_security.h include/grpc/grpc_posix.h include/grpc/load_reporting.h include/grpc/grpc.h include/grpc/census.h include/grpc/impl/ include/grpc/impl/codegen/ include/grpc/impl/codegen/atm_windows.h include/grpc/impl/codegen/atm.h include/grpc/impl/codegen/slice.h include/grpc/impl/codegen/status.h include/grpc/impl/codegen/port_platform.h include/grpc/impl/codegen/connectivity_state.h include/grpc/impl/codegen/sync_windows.h include/grpc/impl/codegen/sync.h include/grpc/impl/codegen/gpr_slice.h include/grpc/impl/codegen/grpc_types.h include/grpc/impl/codegen/atm_gcc_sync.h include/grpc/impl/codegen/exec_ctx_fwd.h include/grpc/impl/codegen/sync_posix.h include/grpc/impl/codegen/compression_types.h include/grpc/impl/codegen/fork.h include/grpc/impl/codegen/byte_buffer_reader.h include/grpc/impl/codegen/atm_gcc_atomic.h include/grpc/impl/codegen/propagation_bits.h include/grpc/impl/codegen/sync_generic.h include/grpc/impl/codegen/byte_buffer.h include/grpc/impl/codegen/gpr_types.h include/grpc/impl/codegen/sync_custom.h include/grpc++/ include/grpc++/resource_quota.h include/grpc++/health_check_service_interface.h include/grpc++/support/ include/grpc++/support/string_ref.h include/grpc++/support/slice.h include/grpc++/support/status.h include/grpc++/support/time.h include/grpc++/support/stub_options.h include/grpc++/support/async_unary_call.h include/grpc++/support/error_details.h include/grpc++/support/status_code_enum.h include/grpc++/support/config.h include/grpc++/support/channel_arguments.h include/grpc++/support/async_stream.h include/grpc++/support/byte_buffer.h include/grpc++/support/sync_stream.h include/grpc++/completion_queue.h include/grpc++/create_channel_posix.h include/grpc++/server_posix.h include/grpc++/server_builder.h include/grpc++/server_context.h include/grpc++/security/ include/grpc++/security/auth_metadata_processor.h include/grpc++/security/server_credentials.h include/grpc++/security/auth_context.h include/grpc++/security/credentials.h include/grpc++/generic/ include/grpc++/generic/generic_stub.h include/grpc++/generic/async_generic_service.h include/grpc++/create_channel.h include/grpc++/client_context.h include/grpc++/server.h include/grpc++/channel.h include/grpc++/ext/ include/grpc++/ext/proto_server_reflection_plugin.h include/grpc++/ext/health_check_service_server_builder_option.h include/grpc++/impl/ include/grpc++/impl/call.h include/grpc++/impl/channel_argument_option.h include/grpc++/impl/client_unary_call.h include/grpc++/impl/rpc_method.h include/grpc++/impl/serialization_traits.h include/grpc++/impl/method_handler_impl.h include/grpc++/impl/rpc_service_method.h include/grpc++/impl/service_type.h include/grpc++/impl/server_builder_plugin.h include/grpc++/impl/codegen/ include/grpc++/impl/codegen/string_ref.h include/grpc++/impl/codegen/proto_utils.h include/grpc++/impl/codegen/slice.h include/grpc++/impl/codegen/call.h include/grpc++/impl/codegen/status.h include/grpc++/impl/codegen/client_unary_call.h include/grpc++/impl/codegen/rpc_method.h include/grpc++/impl/codegen/time.h include/grpc++/impl/codegen/metadata_map.h include/grpc++/impl/codegen/completion_queue.h include/grpc++/impl/codegen/serialization_traits.h include/grpc++/impl/codegen/stub_options.h include/grpc++/impl/codegen/server_interface.h include/grpc++/impl/codegen/method_handler_impl.h include/grpc++/impl/codegen/core_codegen.h include/grpc++/impl/codegen/rpc_service_method.h include/grpc++/impl/codegen/service_type.h include/grpc++/impl/codegen/config_protobuf.h include/grpc++/impl/codegen/completion_queue_tag.h include/grpc++/impl/codegen/async_unary_call.h include/grpc++/impl/codegen/server_context.h include/grpc++/impl/codegen/status_code_enum.h include/grpc++/impl/codegen/security/ include/grpc++/impl/codegen/security/auth_context.h include/grpc++/impl/codegen/create_auth_context.h include/grpc++/impl/codegen/config.h include/grpc++/impl/codegen/core_codegen_interface.h include/grpc++/impl/codegen/channel_interface.h include/grpc++/impl/codegen/call_hook.h include/grpc++/impl/codegen/async_stream.h include/grpc++/impl/codegen/byte_buffer.h include/grpc++/impl/codegen/client_context.h include/grpc++/impl/codegen/sync_stream.h include/grpc++/impl/codegen/grpc_library.h include/grpc++/impl/grpc_library.h include/grpc++/impl/server_initializer.h include/grpc++/impl/server_builder_option.h include/grpc++/alarm.h include/grpc++/grpc++.h include/google/ include/google/protobuf/ include/google/protobuf/util/ include/google/protobuf/util/json_util.h include/google/protobuf/util/delimited_message_util.h include/google/protobuf/util/field_mask_util.h include/google/protobuf/util/field_comparator.h include/google/protobuf/util/message_differencer.h include/google/protobuf/util/time_util.h include/google/protobuf/util/type_resolver.h include/google/protobuf/util/type_resolver_util.h include/google/protobuf/map.h include/google/protobuf/timestamp.pb.h include/google/protobuf/source_context.pb.h include/google/protobuf/generated_message_util.h include/google/protobuf/struct.proto include/google/protobuf/metadata.h include/google/protobuf/map_entry_lite.h include/google/protobuf/dynamic_message.h include/google/protobuf/unknown_field_set.h include/google/protobuf/metadata_lite.h include/google/protobuf/empty.proto include/google/protobuf/map_entry.h include/google/protobuf/io/ include/google/protobuf/io/zero_copy_stream_impl.h include/google/protobuf/io/printer.h include/google/protobuf/io/zero_copy_stream.h include/google/protobuf/io/strtod.h include/google/protobuf/io/tokenizer.h include/google/protobuf/io/coded_stream.h include/google/protobuf/io/zero_copy_stream_impl_lite.h include/google/protobuf/service.h include/google/protobuf/descriptor.h include/google/protobuf/map_type_handler.h include/google/protobuf/message_lite.h include/google/protobuf/wire_format_lite.h include/google/protobuf/any.proto include/google/protobuf/generated_message_table_driven.h include/google/protobuf/stubs/ include/google/protobuf/stubs/bytestream.h include/google/protobuf/stubs/atomicops_internals_arm_qnx.h include/google/protobuf/stubs/logging.h include/google/protobuf/stubs/template_util.h include/google/protobuf/stubs/fastmem.h include/google/protobuf/stubs/status.h include/google/protobuf/stubs/hash.h include/google/protobuf/stubs/once.h include/google/protobuf/stubs/callback.h include/google/protobuf/stubs/atomicops_internals_arm64_gcc.h include/google/protobuf/stubs/atomicops_internals_generic_gcc.h include/google/protobuf/stubs/atomicops_internals_solaris.h include/google/protobuf/stubs/atomicops_internals_x86_gcc.h include/google/protobuf/stubs/atomicops_internals_x86_msvc.h include/google/protobuf/stubs/singleton.h include/google/protobuf/stubs/stringpiece.h include/google/protobuf/stubs/platform_macros.h include/google/protobuf/stubs/atomicops_internals_ppc_gcc.h include/google/protobuf/stubs/shared_ptr.h include/google/protobuf/stubs/atomicops_internals_mips_gcc.h include/google/protobuf/stubs/stl_util.h include/google/protobuf/stubs/atomicops.h include/google/protobuf/stubs/atomicops_internals_tsan.h include/google/protobuf/stubs/type_traits.h include/google/protobuf/stubs/atomic_sequence_num.h include/google/protobuf/stubs/scoped_ptr.h include/google/protobuf/stubs/atomicops_internals_arm_gcc.h include/google/protobuf/stubs/atomicops_internals_power.h include/google/protobuf/stubs/common.h include/google/protobuf/stubs/macros.h include/google/protobuf/stubs/atomicops_internals_generic_c11_atomic.h include/google/protobuf/stubs/mutex.h include/google/protobuf/stubs/port.h include/google/protobuf/stubs/casts.h include/google/protobuf/arenastring.h include/google/protobuf/compiler/ include/google/protobuf/compiler/plugin.pb.h include/google/protobuf/compiler/python/ include/google/protobuf/compiler/python/python_generator.h include/google/protobuf/compiler/plugin.proto include/google/protobuf/compiler/importer.h include/google/protobuf/compiler/javanano/ include/google/protobuf/compiler/javanano/javanano_generator.h include/google/protobuf/compiler/csharp/ include/google/protobuf/compiler/csharp/csharp_names.h include/google/protobuf/compiler/csharp/csharp_generator.h include/google/protobuf/compiler/parser.h include/google/protobuf/compiler/js/ include/google/protobuf/compiler/js/js_generator.h include/google/protobuf/compiler/js/well_known_types_embed.h include/google/protobuf/compiler/cpp/ include/google/protobuf/compiler/cpp/cpp_generator.h include/google/protobuf/compiler/php/ include/google/protobuf/compiler/php/php_generator.h include/google/protobuf/compiler/java/ include/google/protobuf/compiler/java/java_generator.h include/google/protobuf/compiler/java/java_names.h include/google/protobuf/compiler/ruby/ include/google/protobuf/compiler/ruby/ruby_generator.h include/google/protobuf/compiler/command_line_interface.h include/google/protobuf/compiler/objectivec/ include/google/protobuf/compiler/objectivec/objectivec_helpers.h include/google/protobuf/compiler/objectivec/objectivec_generator.h include/google/protobuf/compiler/plugin.h include/google/protobuf/compiler/code_generator.h include/google/protobuf/wrappers.pb.h include/google/protobuf/extension_set.h include/google/protobuf/reflection.h include/google/protobuf/api.proto include/google/protobuf/api.pb.h include/google/protobuf/generated_enum_reflection.h include/google/protobuf/message.h include/google/protobuf/descriptor.pb.h include/google/protobuf/map_field_inl.h include/google/protobuf/type.proto include/google/protobuf/text_format.h include/google/protobuf/wire_format_lite_inl.h include/google/protobuf/any.pb.h include/google/protobuf/reflection_ops.h include/google/protobuf/descriptor_database.h include/google/protobuf/wrappers.proto include/google/protobuf/arena_impl.h include/google/protobuf/wire_format.h include/google/protobuf/map_field_lite.h include/google/protobuf/any.h include/google/protobuf/duration.pb.h include/google/protobuf/struct.pb.h include/google/protobuf/timestamp.proto include/google/protobuf/field_mask.pb.h include/google/protobuf/arena.h include/google/protobuf/field_mask.proto include/google/protobuf/duration.proto include/google/protobuf/empty.pb.h include/google/protobuf/repeated_field.h include/google/protobuf/has_bits.h include/google/protobuf/source_context.proto include/google/protobuf/descriptor.proto include/google/protobuf/map_field.h include/google/protobuf/type.pb.h include/google/protobuf/generated_enum_util.h include/google/protobuf/generated_message_reflection.h

[cyriltovena]

We have the exact same version of docker but I'm 14.04 trusty for WSL, I've asked my organisation to update my computer. Are you using hyper-v ?

Should we get a slack channel ?

[markmandel]

Yep, I am using Hyper-V!

We should sort out some kind of chat next week when we're both back at actual work 😉

I found a bug with part of the development process - submitting now as well, will cc you.

[cyriltovena]

I used an internal repository to avoid the build and the gamecontroller worked on minikube in hyper-v. So looked like all the pieces are working for development purpose on windows, it's just a matter of bringing that all together now.

[markmandel]

Great! The fix for Agon just running on Minikube was pretty straight forward once I got into it - but I definitely want to have it so that people can also build Agon on Windows with Minikube too. I'll likely ping you about that guide to setting it up with Hyper-V.

[markmandel]

Doing some more work on this - slowly getting somewhere with minikube + wsl. But early days - and WSL has to be run as administrator. You can see it in this branch.

It also has the OS specific include setup within the make file, to allow for OS specific configuration.

[cyriltovena]

you didn't get any issue while setting up minikube on windows ? Are you using an external vswitch ?

How does it works if your minikube_home is not mounted using the common mount and you reference $CERT_PATH ? see https://github.com/googleprivate/agon/blob/c39ef2044c2788a883115e9163facccbf76f469d/build/includes/windows.mk#L31

[markmandel]

I didn't use an external switch - and almost everything worked. The only thing that didn't was the minikube.exe mount command, but that actually isn't 100% necessary, it just means you can't build directly on minikube, but you can still do a build locally, then push, which is fine, it just needs to be documented. Otherwise, this seems to be working. I'll clean it up tomorrow, write some docs (and make sure I didn't break anything on Linux), and file a PR.

On the second question (if I understand correctly), the minikube certs are mounted into the Docker container, so that they are accessible. The location of those certs are determined through parsing minikube docker-env to get the appropriate locations. Does that answer the question?

[markmandel]

Oh (almost forgot), once piece of secret sauce I did run into. If you want to run minikube under WSL, you have to run WSL as Administrator, otherwise it just won't work.

The Makefile does do a lot of switching paths from Windows->Linux (and back in a few cases), as well as setting up the Kubernetes config manually as well.

It's not 100% straightforward, but it's automated now.

[cyriltovena]

Ok got it you always use kubectl via docker run, so you're re-mounting the same config and you're using a different user so no conflicts with windows kubectl usage (different path).

[markmandel]

:+1: Ideally, I'd like it if I could actually mount the WSL's ~/.kube into the container (maybe a simlink?) so if people want to use kubectl locally they are also able, but this at least gets things started.