search

googleforgames / agones

Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes
https://agones.dev
Apache License 2.0
6.09k stars 810 forks source link

Running windows game server #54

Open cyriltovena opened 6 years ago

cyriltovena commented 6 years ago

Game production usually works on windows first then port to linux eventually at the end of the development. Having windows support would help the adoption rate.

What does it takes to run a windows game server ?

Testing will be difficult as windows support is still in beta since k8s 1.5 but apparently greatly improve in 1.9.

Documentation : https://kubernetes.io/docs/getting-started-guides/windows/ http://blog.kubernetes.io/2017/09/windows-networking-at-parity-with-linux.html https://github.com/kubernetes/community/tree/master/sig-windows https://docs.microsoft.com/en-us/windows-server/get-started/whats-new-in-windows-server-1709

markmandel commented 6 years ago

It looks like 1.9 support for Windows is much better than previous - I would suggest only trying it in a 1.9 install.

cyriltovena commented 6 years ago

http://blog.kubernetes.io/2017/09/windows-networking-at-parity-with-linux.html this was shared by @alexandrem and if you look at the bottom table you'll see that only 1709 support multiple container in a pod.

markmandel commented 6 years ago

Sorry, I wasn't clear - my thought was that the blog post is pre-1.9, so it may be out of date.

Looking at the Known Limitations Section, which is the 1.9 docs, this is not listed as an issue - where I figure it should be. Hence I figured it makes sense to check, either because it's a bad documentation bug (and we could file a PR), or work has been done to resolve the issue on Windows Server 2016.

cyriltovena commented 6 years ago

You're right and we need to test but from my understanding it's independent from kubernetes development but more an issue of windows server networking and this has been fixed in 1709 version so for me as soon as you don't use an updated version it won't work.

What will be really a problem is testing, so far I don't see any other way than having a real cluster. Because minikube support only linux container and the docker for windows team is apparently going down the minikube road.

markmandel commented 6 years ago

Before testing - would be faster to just drop an email to https://groups.google.com/forum/#!forum/kubernetes-sig-windows :smile: and see if it's a documentation bug or not.

The other part I'm curious about is the state of hostPort support.

markmandel commented 6 years ago

Some more content released today: http://blog.kubernetes.io/2018/01/kubernetes-v19-beta-windows-support.html

cyriltovena commented 6 years ago

https://blog.docker.com/2017/11/docker-for-windows-17-11/

Because there’s only one Docker daemon, and because that daemon now runs on Windows, it will soon be possible to run Windows and Linux Docker containers side-by-side, in the same networking namespace. This will unlock a lot of exciting development and production scenarios for Docker users on Windows.

Possibly no need to change anything in the code as the sidecar would run on linux and the gameserver on windows, this would also be interesting to explore.

markmandel commented 5 years ago

Today - there is an update! https://kubernetes.io/blog/2019/03/25/kubernetes-1-14-release-announcement/

cyriltovena commented 5 years ago

Kind issue to look at: https://github.com/kubernetes-sigs/kind/issues/410

markmandel commented 4 years ago

Big news!!!

https://cloud.google.com/kubernetes-engine/docs/release-notes?hl=en

The ability to create clusters with node pools running Microsoft Windows Server is now in Beta. This feature is currently only available in the Rapid release channel.

EricFortin commented 4 years ago

I have taken a stab at this in the past few days. I somewhat succeeded and thought I would recap my work in this issue so that we may be able to make it happen.

Disclaimer: I was mainly interested in seeing if this could even work. I hacked a few things and work around others.

Building sidecar and simple udp images for Windows

SDK

Here's an example of my final Dockerfile:

# Go runtime methods link to some Windows DLLs that are not in nanoserver. See https://github.com/StefanScherer/dockerfiles-windows/pull/197 for the workaround implemented here.
# Using multi stage build to copy it and still keep a relatively small image.
FROM mcr.microsoft.com/windows/servercore:1909 as core
WORKDIR /app

FROM mcr.microsoft.com/windows/nanoserver:1909

COPY --from=core /windows/system32/netapi32.dll /windows/system32/netapi32.dll

WORKDIR /app
COPY ./sdk-server.windows.amd64.exe /app/sdk-server.exe
ENTRYPOINT ["/app/sdk-server.exe"]

Windows containers images need to be build on the same OS that they are targetting. When trying to build on my machine(1903) with a FROM windows/nanoserver:1909 I had an error so I ultimately built my images on a VM.

simple-udp

I used simple-udp to test as this is simple and can be cross-compiled. I used this command from Agones clone:

docker run --rm -e "GOOS=windows" -e "GOARCH=amd64" -e "GO111MODULE=on" -w /go/src/agones.dev/agones -v /f/go/src/agones.dev/agones/build//.config/gcloud:/root/.config/gcloud -v ~/.kube/:/root/.kube -v ~/.helm:/root/.helm -v /f/go/src/agones.dev/agones:/go/src/agones.dev/agones -v /f/go/src/agones.dev/agones/build//.gomod:/go/pkg/mod -v /f/go/src/agones.dev/agones/build//.gocache:/root/.cache/go-build agones-build:c16b1f68c7 go build -mod=vendor \
        -o /go/src/agones.dev/agones/examples/simple-udp/simple-udp.windows.amd64.exe  -ldflags "-X agones.dev/agones/pkg.Version=1.6.0-7988111" agones.dev/agones/examples/simple-udp

I then wrote a similar Dockerfile as the previous one to build the image.

Cluster creation

I followed GKE's documentation to create a cluster to host Agones with Windows node. It seems we should be able to run in 1.15 but when I tried with this version, I couldn't add the Windows node pool so I reverted to use 1.16 which at the time of writing gave me a 1.16.8-gke15. I used those commands to create the cluster:

gcloud container clusters create special-cluster \
    --enable-ip-alias \
    --num-nodes=1 \
    --machine-type=n1-standard-4 \
    --cluster-version=1.16

gcloud container node-pools create win-node-pool \
    --cluster=special-cluster \
    --image-type=WINDOWS_SAC \
    --no-enable-autoupgrade \
    --metadata disable-legacy-endpoints=true \
    --machine-type=n1-standard-4 \
    --num-nodes=1

gcloud container node-pools create agones-system \
  --cluster=special-cluster \
  --no-enable-autoupgrade \
  --metadata disable-legacy-endpoints=true \
  --node-taints agones.dev/agones-system=true:NoExecute \
  --node-labels agones.dev/agones-system=true \
  --num-nodes=1 \
  --machine-type=n1-standard-4

Agones installation

We need to replace the SDK image that default install use since it is the one build as a linux based container. Helm has the settings to overwrite. Unfortunately, Agones use a single repo url and only the name and tags can be replaced. We can then either host all images in an external repo and replace the image repo settings or we can cheat a little(which I did but it'd be better not to)

Here's how I installed Agones: helm install --set agones.image.sdk.name=agones-sdk-win --set agones.image.sdk.tag=1909 --name my-release --namespace agones-system agones/agones

This means the controller will insert sidecar with image at gcr.io/agones-images/agones-sdk-win:1909. On my side, since I built the image from the actual cluster's Windows node, I simply had to tag it like so. It worked in my favor since Kubernetes' default PullPolicy is IfNotPresent. This isn't great(it only works on single node node pool) and next time I will go with hosting them all in our repo.

Deploying game servers

I used Agones' example and added a node selector to target Windows machine. Here's the content of my fleet.yaml:

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: simple-udp
spec:
  replicas: 2
  template:
    spec:
      ports:
      - name: default
        containerPort: 7654
      template:
        spec:
          nodeSelector:
            kubernetes.io/os: windows
          containers:
          - name: simple-udp
            image: <redacted>
            resources:
              requests:
                memory: "64Mi"
                cpu: "100m"
              limits:
                memory: "64Mi"
                cpu: "200m"

After that, I had 2 game servers running in my cluster. Allocation works too.

$k get gs
NAME                     STATE   ADDRESS         PORT   NODE                                              AGE
simple-udp-wk2rq-9r6b7   Ready   <redacted>      7081   gke-special-cluster-win-node-pool-568b60bb-dh14   13h
simple-udp-wk2rq-vqjvq   Ready   <redacted>      7145   gke-special-cluster-win-node-pool-568b60bb-dh14   21h

End of good news

I was not able to contact my game server. So I did some investigation to diagnose: Firewall rules is correctly setup as when running the sdk in local mode and simple-udp directly on Windows VM, I was able to contact the server.

To run in GKE with Windows node, we have to enable ip alias which means each pod have their own IPs. Kubernetes then need to setup routing when using host ports. So I tested from within the cluster if I was able to reach game servers. Here are some results:

That means the game server hosting in Windows container is working but there are issues with the networking setup. After snooping around on the Windows VM, I found out that GKE uses L2bridge type network driver and specially win-bridge CNI plugin. It looks like this plugin supports hostPort mapping where it would ask the Host Network Service(HNS) to configure some routing rules. At this point, my Windows Networking knowledge is coming a bit short and I can't really debug CNI integration in GKE.

I tested a standard cluster(only Container optimized OS VMs) and it worked so it point to a shortcoming in the Windows CNI implementation.

To be continued ...

cyriltovena commented 4 years ago

Here's how I installed Agones: helm install --set agones.image.sdk.name=agones-sdk-win --set agones.image.sdk.tag=1909 --name my-release --namespace agones-system agones/agones

Would be great to be able to specify the SDK image via an annotation, this way you don't need to reinstall agones and you can run different sdk version.

Nice experiment.

roberthbailey commented 4 years ago

We need to replace the SDK image that default install use since it is the one build as a linux based container. Helm has the settings to overwrite. Unfortunately, Agones use a single repo url and only the name and tags can be replaced. We can then either host all images in an external repo and replace the image repo settings or we can cheat a little(which I did but it'd be better not to)

On GKE (I think this is also true for other providers) you have to have at least one node pool running linux VMs for system containers even if your are running your workloads on Windows. So you could just run the linux agones controllers next to the other linux Kubernetes controllers.

edit: I just re-read this and realized you are talking about the sidecar.....

For that, I think we need to add some flexibility into Agones so that it can launch both windows and linux sidecars based on some sort of tag on the gameserver / fleet.

roberthbailey commented 4 years ago

I had been wondering about hostPort support. I read through the GKE and Kubernetes docs and didn't see a clear answer as to whether (or how well) they would work. Maybe someone on the GKE windows or GKE networking team can help. We will reach out internally.

WeetA34 commented 4 years ago

Hello, I've setup a local cluster on my windows workstation with 3 hyper-VM vm. I deployed kub with rancher 2.4.4 1st try with flannel vxlan but i encounter a big which avoid windows container to reach kube api (10.43.0.1). It's a known issue which should have been fixed in kube 1.18.1. So, i rebuilt the cluster with flannel l2bridge. I'm stuck at the same point as Eric. The gameserver is running and is healthy but i'm not able to connect on the hostPort from another machine located on the same hyper-v network. No issue with the simple-tcp linux. I've checked firewall on the windows server core 1903 node. It seems ok. I'm still investigating. Regards Stéphane

WeetA34 commented 4 years ago

I ran almost all commands present in https://github.com/microsoft/SDN/tree/master/Kubernetes/windows/debug/ scripts I didn't find the expected port

WeetA34 commented 4 years ago

Same issue without agones, unable to connect to IIS container launched with the following command: kubectl run iis --namespace default --restart='Never' --port 80 --hostport 8000 --image=mcr.microsoft.com/windows/servercore/iis:windowsservercore-1903 no issue with a simple docker run: docker run -d --rm --name iis -p 8000:80 mcr.microsoft.com/windows/servercore/iis:windowsservercore-1903

markmandel commented 4 years ago

Started a thread as well in #sig-windows on K8s slack: https://kubernetes.slack.com/archives/C0SJ4AFB7/p1591381173181100

markmandel commented 4 years ago

Based on the conversation in the #sig-windows thread, hostPort is not currently supported.

I've filed/updated two tickets (see links above). and been told to check in in a week, to see if they can find some resources to work on hostPort support:

daschott  14 minutes ago
Yes, that would be a good issue to upvote, and include the information. I would considering raising a Windows issue on CNI plugins as well. Let me try to see internally if anyone on our team has cycles to pick this up. Please follow up again in a week if you haven't heard back by then.
markmandel commented 4 years ago

Possible good news! https://github.com/containernetworking/plugins/releases/tag/v0.8.6 supports portMappings! So it may be a case of an updated CNI, and ensuring:

    "capabilities": {
        "dns": true,
        "portMappings":  true
    }

has been added to the CNI config.

markmandel commented 4 years ago

Looks like we do have it enabled on GCE: https://github.com/kubernetes/kubernetes/blob/release-1.15/cluster/gce/windows/k8s-node-setup.psm1#L807

But need the CNI version is not up to 0.8.6 (which came out 23 days ago, so no surprise)

WeetA34 commented 4 years ago

Hello Mark, i already created an flannel image with 0.8.6 but i was not aware about portMappings. i can now access iis when running iis windows container with the following command: kubectl run iis --namespace default --restart='Never' --port 80 --hostport 8000 --image=mcr.microsoft.com/windows/servercore/iis:windowsservercore-1903 Now i'm facing an issue creating the gs resource due to kubernetes 1.18.x server side apply.

kubectl create -f gs-simple-tcp-win.yaml

The GameServer "simple-tcp-win-69qbp" is invalid: metadata.managedFields.fieldsType: Invalid value: "": must be `FieldsV1`

Thank you

WeetA34 commented 4 years ago

Hello again i just rebuilt the entire cluster with kubernetes 1.17.6 It works fine now :) flannel type: host-gw (l2bridge) windows node & image build: windows server 1903

Thank you again Mark for pointing to the new CNi plugin portMappings attributes It's documented here : https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems#hostport-publishing-is-not-working

$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION       CONTAINER-RUNTIME
kub2lin01      Ready    <none>   34m   v1.17.6   172.31.0.59     <none>        Ubuntu 18.04.4 LTS        4.15.0-101-generic   docker://19.3.11
kub2master01   Ready    master   38m   v1.17.6   172.31.10.248   <none>        Ubuntu 18.04.4 LTS        4.15.0-101-generic   docker://19.3.11
kub2win01      Ready    <none>   12m   v1.17.6   10.244.2.2      <none>        Windows Server Standard   10.0.18362.836       docker://19.3.5

$ kubectl get gameservers
NAME                   STATE   ADDRESS      PORT   NODE        AGE
simple-tcp-win-mpgsc   Ready   10.244.2.2   7612   kub2win01   3m56s

$ kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
simple-tcp-win-mpgsc   2/2     Running   5          4m24s   10.244.2.5   kub2win01   <none>           <none>

Note: The windows node is registered with its flannel IP instead of its primary IP. Kubernetes Windows Node:

C:\>hostname
KUB2WIN01

C:\>ipconfig

Windows IP Configuration

Ethernet adapter vEthernet (Ethernet) 2:

   Connection-specific DNS Suffix  . : mshome.net
   Link-local IPv6 Address . . . . . : fe80::8c2a:c459:b65b:5fa7%14
   IPv4 Address. . . . . . . . . . . : 172.31.10.83
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.31.0.1

Ethernet adapter vEthernet (cbr0_ep):

   Connection-specific DNS Suffix  . : mshome.net
   Link-local IPv6 Address . . . . . : fe80::9848:7de1:9f17:a440%15
   IPv4 Address. . . . . . . . . . . : 10.244.2.2
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.244.2.1

Ethernet adapter vEthernet (nat):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::181c:d022:213c:c96%9
   IPv4 Address. . . . . . . . . . . : 192.168.160.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :

Ethernet adapter vEthernet (ae554c2617bbdc9):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::d008:e1bc:7902:e1b7%16
   IPv4 Address. . . . . . . . . . . : 172.21.192.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :

Test from an Alpine VM on the same subnet:

$  hostname
alpine

$ ip addr show eth0 | grep 'inet '
    inet 172.31.8.45/20 scope global eth0

$ nc 172.31.10.83 7612
hello toto
ACK: hello toto

I built the following images on my windows node:

REPOSITORY                             TAG                      IMAGE ID            CREATED             SIZE
sigwindowstools/flannel                0.12.0-1903              af8d76c4c777        33 minutes ago      5.21GB
golang                                 windowsservercore-1903   25eabeca5478        35 minutes ago      5.54GB
sigwindowstools/kube-proxy             1.17.6-1903              874e362d6bd9        40 minutes ago      5.19GB
mcr.microsoft.com/k8s/core/pause       1.2.0-1903               995bb8fbeb6d        4 hours ago         261MB
simple-tcp-win                         0.0.2                    b46163501dbd        44 hours ago        269MB
gcr.io/agones-images/agones-sdk        1.6.0                    5c95ad75fdc2        2 days ago          296MB
WeetA34 commented 4 years ago

FYI, flannel windows daemonset pod doesn't restart after windows node reboot. I created the following issue to kubernetes-sigs: https://github.com/kubernetes-sigs/sig-windows-tools/issues/94

WeetA34 commented 4 years ago

Successful test with a fleet:

kubemaster01$ kubectl create -f fleet.yaml
fleet.agones.dev/fleet-simple-tcp-win created

kubemaster01$ kubectl get gameservers
NAME                               STATE   ADDRESS      PORT   NODE        AGE
fleet-simple-tcp-win-c2bqh-wrt7c   Ready   10.244.2.2   7320   kub2win01   7s
fleet-simple-tcp-win-c2bqh-zz56w   Ready   10.244.2.2   7135   kub2win01   7s

kubemaster01$ kubectl scale fleet fleet-simple-tcp-win --replicas=10
fleet.agones.dev/fleet-simple-tcp-win scaled

kubemaster01$ kubectl get gameservers
NAME                               STATE   ADDRESS      PORT   NODE        AGE
fleet-simple-tcp-win-c2bqh-884h4   Ready   10.244.2.2   7303   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-8wcjj   Ready   10.244.2.2   7148   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-h4m4h   Ready   10.244.2.2   7123   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-knr9f   Ready   10.244.2.2   7011   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-sfzfk   Ready   10.244.2.2   7200   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-swss8   Ready   10.244.2.2   7105   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-wrt7c   Ready   10.244.2.2   7320   kub2win01   4m29s
fleet-simple-tcp-win-c2bqh-x67xv   Ready   10.244.2.2   7456   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-zbk7f   Ready   10.244.2.2   7855   kub2win01   3m29s
fleet-simple-tcp-win-c2bqh-zz56w   Ready   10.244.2.2   7135   kub2win01   4m29s

kubemaster01$ kubectl get gameservers -o jsonpath='{.items[*].spec.ports[0].hostPort}'
7303 7148 7123 7011 7200 7105 7320 7456 7855 7135

alpine$ for port in 7303 7148 7123 7011 7200 7105 7320 7456 7855 7135; do echo "hello gs listening on port ${port}" | nc -w 1 172.31.10.83 $port; done
ACK: hello gs listening on port 7303

ACK: hello gs listening on port 7148

ACK: hello gs listening on port 7123

ACK: hello gs listening on port 7011

ACK: hello gs listening on port 7200

ACK: hello gs listening on port 7105

ACK: hello gs listening on port 7320

ACK: hello gs listening on port 7456

ACK: hello gs listening on port 7855

ACK: hello gs listening on port 7135
WeetA34 commented 4 years ago

Now i have to fix the problem of the wrong windows node ip. I guess game clients will not be able to connect to game servers because the public IP of the windows node is not known on kubernetes.

WeetA34 commented 4 years ago

Found it. You can either add --node-ip <YOUR_NODE_IP> to:

Then, restart kubelet service.

WeetA34 commented 4 years ago

Hello,

it works fine with UDP too

$ kubectl create -f fleet-udp.yaml
fleet.agones.dev/fleet-simple-udp-win created

$ kubectl get fleet
NAME                   SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY   AGE
fleet-simple-udp-win   Packed       2         2         0           2       2m54s

$ kubectl get gs
NAME                               STATE   ADDRESS        PORT   NODE        AGE
fleet-simple-udp-win-rzpgd-jzfh6   Ready   172.31.10.83   7601   kub2win01   19s
fleet-simple-udp-win-rzpgd-xs68b   Ready   172.31.10.83   7445   kub2win01   19s

alpine:~$ for port in 7601 7445; do echo "hello gs listening on port ${port}" | nc -w 1 -u 172.31.10.83 $port; done
ACK: hello gs listening on port 7601
ACK: hello gs listening on port 7445
WeetA34 commented 4 years ago

Hello of the day :)

I built a Windows Server core 1903 image with Xonotic Game server. I successfully created a fleet with this image. I was able to connect with the game to the Game server. below some logs of the Game server

Server using port 26000
>>> Found 'listening' statement: 4
>>> Moving to READY: Server listening on address 0.0.0.0:26000
Server listening on address [0:0:0:0:0:0:0:0]:26000
Server listening on address 0.0.0.0:26000
Server listening on address [0:0:0:0:0:0:0:0]:26000
Error response from keygen server: Not allowed
Authenticated connection to 172.31.0.1:59722 has been established: client is glZQY................=@....., I am ow.....@.........
WeetA is connecting...
WeetA connected
WeetA connected
WeetA is now spectating
[BOT]Thunderstorm connected
[BOT]Discovery connected
[BOT]Mystery connected
[BOT]Delirium connected
unconnected changed name to [BOT]Thunderstorm
unconnected changed name to [BOT]Discovery
unconnected changed name to [BOT]Mystery
unconnected changed name to [BOT]Delirium
Receiving player stats failed: -404
[BOT]Discovery picked up Strength
[BOT]Discovery drew first blood!
[BOT]Delirium was gunned down by [BOT]Discovery's Shotgun
[BOT]Delirium got too close to [BOT]Discovery's Mortar grenade
[BOT]Thunderstorm slapped [BOT]Mystery around a bit with a large Shotgun
[BOT]Thunderstorm blew themself up with their own Mortar
[BOT]Discovery made a TRIPLE FRAG!
[BOT]Mystery was telefragged by [BOT]Discovery
[BOT]Discovery slapped [BOT]Delirium around a bit with a large Shotgun
[BOT]Discovery unlocked RAGE!
[BOT]Thunderstorm was gunned down by [BOT]Discovery's Shotgun
[BOT]Discovery slapped [BOT]Delirium around a bit with a large Shotgun
[BOT]Delirium ate [BOT]Mystery's rocket
[BOT]Thunderstorm was riddled full of holes by [BOT]Mystery's Machine Gun
[BOT]Mystery picked up Strength
[BOT]Mystery made a TRIPLE FRAG!
[BOT]Delirium was pummeled by a burst of [BOT]Mystery's Hagar rockets
[BOT]Delirium slapped [BOT]Thunderstorm around a bit with a large Shotgun
Client "WeetA" dropped
WeetA disconnected
Client "[BOT]Delirium" dropped
[BOT]Delirium disconnected
Client "[BOT]Thunderstorm" dropped
[BOT]Thunderstorm disconnected
Client "[BOT]Discovery" dropped
[BOT]Discovery disconnected
Client "[BOT]Mystery" dropped
[BOT]Mystery disconnected
eddiebeazer commented 4 years ago

Here's how I installed Agones: helm install --set agones.image.sdk.name=agones-sdk-win --set agones.image.sdk.tag=1909 --name my-release --namespace agones-system agones/agones

I'm working on trying to replicate the same example since the server I'm using has to be ran on Windows. I got everything working minus the sidecar. How would I go about creating a 1909 version of the current 1.9 sdk?

EricFortin commented 4 years ago

@eddiebeazer Are you talking about the golang binary or the sdk image? What issues did you face with 1.9 particularly? I was able to just use the cross compiled sdk binary when I did my investigation.

eddiebeazer commented 4 years ago

@eddiebeazer Are you talking about the golang binary or the sdk image? What issues did you face with 1.9 particularly? I was able to just use the cross compiled sdk binary when I did my investigation.

@EricFortin The sdk image. I'm creating an unreal server with windows and I followed through the tutorials to adding the Agones BP to main map. When I roll out my fleet config the server image is pulled fine but the Agones image doesn't pull at all when I try to use --set agones.image.sdk.name=agones-sdk-win --set agones.image.sdk.tag=1909.

I'm completely new to Agones and haven't got the chance to dive deep yet in the sdk since I haven't been able to get a basic working example yet. I don't know if I even need the sidecar if the sdk is implemented inside of my unreal server.

Overall what I'm trying to accomplish is to get Open Match, Agones and an Unreal Engine windows server working in GKE. Currently got the server up fine and working on the Agones portion and got stuck with getting the sidecar to work correctly.

If it helps, I'm using the default terraform helm install from the agones repo. I made a local copy of it so I could add these two statements to it

set {
    name  = "agones.image.sdk.name"
    value = "agones-sdk-win"
  }

  set {
    name  = "agones.image.sdk.tag"
    value = "1909"
  }

Besides that, everything else is the same from the repo I copied it from.

EricFortin commented 4 years ago

Gotcha. Agones is not building Windows images at the moment and by default will pull from its own gcr repo. When I made my investigation, I had to build the image myself and I cheated as I built it directly on the node(and tagged it as if it was coming from the Agones repo) where the game server would be running because I didn't want to host it myself.

If you want to build your own image, I detailed the steps I took in my comments: https://github.com/googleforgames/agones/issues/54#issuecomment-637633950

jeremyje commented 4 years ago

To add on what https://github.com/googleforgames/agones/issues/54#issuecomment-637633950 says.

Given that Agones is a Linux centric project it's actually still possible to build Windows container images using DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build as long as you're not using RUN. You need the latest version of docker for this since it requires buildkit. I can confirm this works for docker-in-docker scenarios since Agones does that from what I remember. buildx is kinda weird and you'll most likely have to use --push to push the image immediately after the build.

There's a Google Cloud Builder community project for building on Windows hosts https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/windows-builder.

Lastly, for building Windows images on image hosts. It appears you can do this as long as the host is at or newer than the version under build though I wouldn't press my luck here since there are can be incompatibilities.

WeetA34 commented 4 years ago

Hi, i built 2 new test environments (1 on openstack and 1 on a hyper-v workstation)

the sidecar container doesn't start workers

+ simple-tcp-win-l6qn6 › agones-gameserver-sidecar
+ simple-tcp-win-l6qn6 › simple-tcp-win
simple-tcp-win-l6qn6 simple-tcp-win 2020/09/04 12:47:11 Starting TCP server, listening on port 7654
simple-tcp-win-l6qn6 simple-tcp-win 2020/09/04 12:47:12 Creating SDK instance
simple-tcp-win-l6qn6 simple-tcp-win 2020/09/04 12:47:13 Starting Health Ping
simple-tcp-win-l6qn6 simple-tcp-win 2020/09/04 12:47:13 Marking this server as ready
simple-tcp-win-l6qn6 agones-gameserver-sidecar {"ctlConf":{"Address":"localhost","IsLocal":false,"LocalFile":"","Delay":0,"Timeout":0,"Test":"","TestSdkName":"","GRPCPort":9357,"HTTPPort":9358},"featureGates":"ContainerPortAllocation=true\u0026Example=true\u0026PlayerTracking=false\u0026SDKWatchSendOnExecute=false","message":"Starting sdk sidecar","severity":"info","source":"main","time":"2020-09-04T12:47:12.8533173Z","version":"1.8.0"}
simple-tcp-win-l6qn6 agones-gameserver-sidecar {"gsKey":"default/simple-tcp-win-l6qn6","message":"Created GameServer sidecar","severity":"info","source":"*sdkserver.SDKServer","time":"2020-09-04T12:47:13.1058722Z"}
simple-tcp-win-l6qn6 agones-gameserver-sidecar {"grpcEndpoint":"localhost:9357","message":"Starting SDKServer grpc service...","severity":"info","source":"main","time":"2020-09-04T12:47:13.1109151Z"}
simple-tcp-win-l6qn6 agones-gameserver-sidecar {"httpEndpoint":"localhost:9358","message":"Starting SDKServer grpc-gateway...","severity":"info","source":"main","time":"2020-09-04T12:47:13.1109151Z"}
simple-tcp-win-l6qn6 agones-gameserver-sidecar {"message":"shutting down sdk server","severity":"info","source":"main","time":"2020-09-04T12:47:22.8980415Z"}
simple-tcp-win-l6qn6 agones-gameserver-sidecar {"error":"failed to wait for caches to sync","message":"Could not run sidecar","severity":"fatal","source":"main","time":"2020-09-04T12:47:22.8980415Z"}
simple-tcp-win-l6qn6 simple-tcp-win 2020/09/04 12:47:23 Could not send health ping: could not send Health ping: EOF

Windows containers are not able to reach 10.96.0.0/12 subnet ie from an IIS container:

PS C:\> ping 10.96.0.1
Pinging 10.96.0.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

ping which doesn't work seems expected as i read but curl doesn't work too.

PS C:\> curl.exe -v -k https://10.96.0.1/
*   Trying 10.96.0.1...
* TCP_NODELAY set
* connect to 10.96.0.1 port 443 failed: Timed out
* Failed to connect to 10.96.0.1 port 443: Timed out
* Closing connection 0

i created an issue to kubernetes-sigs/sig-windows-tools because i tried different versions of docker, kubernetes, flannel, cni plugin. I'm still not able to contact the service network (10.96.0.0/12) from windows pods

markmandel commented 4 years ago

simple-tcp-win-l6qn6 agones-gameserver-sidecar {"error":"failed to wait for caches to sync","message":"Could not run sidecar","severity":"fatal","source":"main","time":"2020-09-04T12:47:22.8980415Z"}

Sounds like there was an issue accessing the master? Or maaaaaybe an RBAC issue? i.e. it couldn't query the specific resource?

jeremyje commented 4 years ago

https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#network-modes https://support.microsoft.com/en-us/help/4489899/windows-10-update-kb4489899 might not be installed. Also some network modes are not available.

WeetA34 commented 4 years ago

Yep, it's a communication issue between windows pod and kubernetes Api. i can contact kubernetes Api when i only have 1 master node and 1 windows worker node. If i add 1 linux worker node, it doesn't work anymore. KB is not applicable in Windows 2019 August 2020 update as it's already included. Flannel host-gw (l2bridge) works fine but i can't use it on our openstack environment due to antispoof policies.

edit: i created the following issue https://github.com/kubernetes-sigs/sig-windows-tools/issues/101 I found there are known issues when accessing service within windows pod. I put links in the issue above I'm able to access the kubernetes api server throught the public endpoint https://:6443 instead of https://10.96.0.1

WeetA34 commented 4 years ago

As communication with the kubernetes api server works only with master node public ip, i 'm using a podpreset to overrides the two following environment variables:

apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: overrides-kubernetes-service-envs
spec:
  selector:
    matchLabels:
      agones.dev/role: gameserver
  env:
    - name: "KUBERNETES_SERVICE_HOST"
      value: "192.168.168.2"
    - name: "KUBERNETES_SERVICE_PORT"
      value: "6443"
WeetA34 commented 4 years ago

I rebuilt everything with kubernetes 1.19.0 The kubernetes api connection issue from windows persists. So i implemented the podpreset workaround i described in my previous message. Now, the GS is ready but port Mapping doesn't work :( I tested cniplugin 0.8.6 and 0.87 with portMappings capability set to true.

PS C:\> cat C:\etc\cni\net.d\10-flannel.conf
{
    "name":  "flannel.4096",
    "cniVersion":  "0.3.0",
    "type":  "flannel",
    "capabilities":  {
                         "dns":  true,
                         "portMappings":  true
                     },
    "delegate":  {
                     "type":  "win-overlay",
                     "policies":  [
                                      {
                                          "Name":  "EndpointPolicy",
                                          "Value":  {
                                                        "Type":  "OutBoundNAT",
                                                        "ExceptionList":  [
                                                                              "10.96.0.0/12",
                                                                              "10.244.0.0/16"
                                                                          ]
                                                    }
                                      },
                                      {
                                          "Name":  "EndpointPolicy",
                                          "Value":  {
                                                        "Type":  "ROUTE",
                                                        "DestinationPrefix":  "10.96.0.0/12",
                                                        "NeedEncap":  true
                                                    }
                                      }
                                  ]
                 }
}

PS C:\> C:\opt\cni\bin\win-overlay.exe --version
CNI win-overlay plugin v0.8.6

damn, it seems that portMappings capability has only been added to win-bridge :( https://github.com/containernetworking/plugins/pull/475

WeetA34 commented 3 years ago

Hello @markmandel I'm testing Agones win in GKE. I'm not able to connect to Win Game Servers (UDP or TCP) I see win-bridge version is: CNI win-bridge plugin v0.8.2-gke.0 In cni config file, i see portMappings is set to True. Firewall rules seem OK. I'll check tomorrow if i forgot something. Any idea? Thank you

markmandel commented 3 years ago

How are you testing it?

WeetA34 commented 3 years ago

With nc from remote. I still have a firewall issue. I enabled port 7000-8000 on the windows node itself but it doesn't work. Nothing received from windows node. When i disable firewall profiles, i see a reset sent by the windows node.

markmandel commented 3 years ago

With nc from remote.

Sorry, I've no idea what this means

WeetA34 commented 3 years ago

PortMappings should work on windows with your custom cni plugin, right?

WeetA34 commented 3 years ago

netcat (nc) from my workstation

markmandel commented 3 years ago

Ah gotcha - I mean, how are you testing the that windows binaries etc? You running a custom fork atm?

markmandel commented 3 years ago

Also, how are you configuring your firewall rules?

I assume it's the same as on linux nodes? set a network tag, then apply a VPC firewall rule?

WeetA34 commented 3 years ago

Ha ok :)

I modified your terraform:

then i manually built the following images directly on the windows node:

WeetA34 commented 3 years ago

FYI, i rebooted the windows node to see if something was not broken. It's worse since :( unable to get GS in ready state

{"message":"agones.dev/agones/pkg/client/informers/externalversions/factory.go:119: Failed to list *v1.GameServer: Get \"https://10.28.0.1:443/apis/agones.dev/v1/namespaces/default/gameservers?fieldSelector=metadata.name%3Dsimple-tcp-win-85qbd\u0026limit=500\u0026resourceVersion=0\": dial tcp 10.28.0.1:443: connectex: A socket operation was attempted to an unreachable network.","severity":"error","time":"2020-11-05T09:57:58.3491579Z"}