search

Octops / gameserver-ingress-controller

Automatic Ingress configuration for Game Servers managed by Agones
https://octops.io
Apache License 2.0
68 stars 9 forks source link

Octops causes ingress controller to constantly reload which causes dropped websocket connections #21

Closed jordo closed 2 years ago

jordo commented 2 years ago

Creation of Ingress objects causes a reload of the nginx controller, which ultimately shuts down all nginx worker processes (https://kubernetes.github.io/ingress-nginx/how-it-works/#when-a-reload-is-required). All existing websocket connections that serviced by that controller will eventually be disconnected after worker_shutdown_timeout expires: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-shutdown-timeout

We discovered this after consistently seeing our existing websocket connections that were proxied through the same ingress controller as octops, disconnect at approximate the same time at approximatlely 4min internals (240s is the default worker-shutdown-timeout). This is also an issue with Http connections that keep a socket open via keep-alive as well.

This is documented in a few places: https://github.com/kubernetes/ingress-nginx/issues/6731 https://github.com/kubernetes/ingress-nginx/issues/7115

And a good summary write up below: https://danielfm.me/post/painless-nginx-ingress/#ingress-classes-to-the-rescue

But ultimately a reload of the configuration will cause socket connections to drop eventually. In the first link above, nginx developers expect that the solution to the problem is the client library handles the reconnect. which is obviously a problem when dealing with real-time games all running via websocket through the same ingress controller.

It should be noted that nginx+ (enterprise paid product) does not have this limitation: (https://www.nginx.com/faq/how-does-zero-downtime-configuration-testingreload-in-nginx-plus-work/) https://www.nginx.com/blog/using-nginx-plus-to-reduce-the-frequency-of-configuration-reloads/

jordo commented 2 years ago

Some example logs from our ingress controller when playing on our staging cluster with octops controller enabled. You can see the ingress controller reloads every time a new ingress resource is created:


I0203 19:52:23.697929       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 19:52:24.209096       6 controller.go:163] "Backend successfully reloaded"
I0203 19:52:24.210289       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 19:52:30.457386       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 19:52:30.705136       6 controller.go:163] "Backend successfully reloaded"
I0203 19:52:30.706209       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 19:53:00.434621       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 19:53:00.689898       6 controller.go:163] "Backend successfully reloaded"
I0203 19:53:00.691200       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 20:05:11.867448       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 20:05:12.444033       6 controller.go:163] "Backend successfully reloaded"
I0203 20:05:12.445512       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 20:05:15.134302       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 20:05:15.453845       6 controller.go:163] "Backend successfully reloaded"
I0203 20:05:15.455368       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 20:05:37.013543       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 20:05:37.316147       6 controller.go:163] "Backend successfully reloaded"
I0203 20:05:37.319452       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 20:05:40.346314       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 20:05:40.591616       6 controller.go:163] "Backend successfully reloaded"
I0203 20:05:40.592452       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:30:11.061375       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:30:11.306018       6 controller.go:163] "Backend successfully reloaded"
I0203 21:30:11.307283       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:34:37.777301       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:34:38.357232       6 controller.go:163] "Backend successfully reloaded"
I0203 21:34:38.358759       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:34:41.110120       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:34:41.413161       6 controller.go:163] "Backend successfully reloaded"
I0203 21:34:41.414632       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:35:31.260247       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:35:31.523111       6 controller.go:163] "Backend successfully reloaded"
I0203 21:35:31.526728       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:40:04.060345       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:40:04.601178       6 controller.go:163] "Backend successfully reloaded"
I0203 21:40:04.610565       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:40:07.382857       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:40:07.755838       6 controller.go:163] "Backend successfully reloaded"
I0203 21:40:07.757164       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:40:57.174052       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:40:57.502071       6 controller.go:163] "Backend successfully reloaded"
I0203 21:40:57.505795       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:54:01.241225       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:54:01.972988       6 controller.go:163] "Backend successfully reloaded"
I0203 21:54:01.974597       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:57:53.944692       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:57:54.277236       6 controller.go:163] "Backend successfully reloaded"
I0203 21:57:54.278582       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:58:30.428631       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:58:30.878348       6 controller.go:163] "Backend successfully reloaded"
I0203 21:58:30.880584       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 21:59:31.081475       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 21:59:31.415205       6 controller.go:163] "Backend successfully reloaded"
I0203 21:59:31.416535       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 22:05:01.534419       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 22:05:02.141184       6 controller.go:163] "Backend successfully reloaded"
I0203 22:05:02.143331       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 22:05:37.021182       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 22:05:37.353131       6 controller.go:163] "Backend successfully reloaded"
I0203 22:05:37.354165       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 22:05:40.353361       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 22:05:40.837159       6 controller.go:163] "Backend successfully reloaded"
I0203 22:05:40.838407       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 22:06:37.007837       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 22:06:37.311283       6 controller.go:163] "Backend successfully reloaded"
I0203 22:06:37.312053       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0203 22:09:56.993499       6 controller.go:146] "Configuration changes detected, backend reload required"
I0203 22:09:57.205114       6 controller.go:163] "Backend successfully reloaded"
I0203 22:09:57.206113       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"gaming-cluster-ingress-nginx-controller-69c54ff79f-5xl48", UID:"e6cf6c05-1378-4ea3-a080-ad4a132cbfe8", APIVersion:"v1", ResourceVersion:"70067102", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration```
danieloliveira079 commented 2 years ago

Tks @jordo I will investigate that too.

danieloliveira079 commented 2 years ago

Quick update: The restart is not caused by the octops controller. This is actually a behaviour implemented by the NGINX ingress controller which causes a restart of the process when an Ingress resource is created, updated or deleted. See details on https://kubernetes.github.io/ingress-nginx/how-it-works/#when-a-reload-is-required.

The same problem is not present when using another ingress controller like HAProxy or Contour.

I will keep this issue open while investigating the mentioned alternatives to NGINX Ingress Controller. My goal is to write down a detailed How-To for using another controller together with the Octops Controller.

jordo commented 2 years ago

We have promoted all of ingress traffic through HAProxy ingress controller (as an alternative to open-source nginx ingress controller), and can confirm we no longer receive any socket disconnections on any active connection that is routed through the same reverse proxy (HAProxy now).

jordo commented 2 years ago

And as example, we were able to accomplish feature parity of what nginx provided for our use case via octops prefixes below:

        octops.io/gameserver-ingress-mode: "path"
        octops.io/gameserver-ingress-fqdn: {{ .Values.gameDomain }}
        octops-external-dns.alpha.kubernetes.io/hostname: {{ .Values.gameDomain }}
        octops-external-dns.alpha.kubernetes.io/ttl: "60"
        octops-kubernetes.io/ingress.class: haproxy
        octops-haproxy.org/server-ssl: "true"
jordo commented 2 years ago

controller we are using is haproxytech/kubernetes-ingress

danieloliveira079 commented 2 years ago

Closing this issue due to the update that supports HAProxy.