websocket long live connection reset on configuration reload

[s-parthiban]

Hi Guys, Ours is the real-time audio transcription application running in the Azure Kubernetes Service(AKS). We are using the WebSocket as the communication protocol which is going to be a long live persistence connection. We are frequently adding or remove the host in the ingress resource of k8s, so we need the ingress controller which is going to dynamically reload the configuration with zero downtime. So we planned to use the haproxy-ingress as ingress controller but sadly on reloading the configuration it resets already established long live connections even if 'reload-strategy' is set as 'reusesocket'. We can set the timeout-stop=0s to stop reset the long live connections until the client closes it, but in turn, it will increase the number of processes running in the haproxy-ingress and it may lead to some other problems. The haproxy-ingress is also not recommended it. Our requirement is the haproxy-ingress should reload the frequently changing ingress configuration without disturbing the long live WebSocket connections. is there any configuration or way to achieve it? Our application haproxy-ingress configurations for your reference Command-line configuration

======================= containers:

• args:
  • --configmap=default/haproxy-ingress
  • --default-ssl-certificate=default/tls-secret
  • --ingress-class=haproxy
  • --sort-backends
  • --reload-strategy=reusesocket env:
  • name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name
  • name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace image: quay.io/jcmoraisjr/haproxy-ingress:v0.13.4 Configmap haproxy-ingress configuration ''' apiVersion: v1 data: healthz-port: "10253" nbthread: "4" ssl-certificate: default/tls-secret ssl-redirect: "true" stats-port: "1936" kind: ConfigMap '''

[jcmoraisjr]

hi, it seems you've already added this same question in the slack channel, let's continue this conversation there.

[github-actions[bot]]

This issue got stale and will be closed in 7 days.

[tomklapka]

@jcmoraisjr Hi, I'm interrested in this as well. What are the correct settings to prevent this issue ?

[jcmoraisjr]

There's no way to avoid reloads, but one can instead configure the controller in a way it serves long lived connections as long as possible. There are a few tips I can share:

• use --master-worker, this will enable some controlling features from haproxy, doc here
• consider a custom value to --reload-interval if your users can tolerate a delay to see their changes applied, doc here
• configure global worker-max-reloads to the higher number your ingress nodes can support, doc here
• configure globals timeout-stop and timeout-tunnel to an insane value, e.g. 24h, doc here, leaving worker-max-reloads to do its job

I'm reopening this issue as a doc one, so we'll remember to add these infos somewhere in the doc.

[tomklapka]

@jcmoraisjr thanks for the tips, most of them I had already implemented, but still, when we update the app image (and therefore rolling update is iniciated) behing the haproxy existing backend config it always reloads the config and terminates all sessions connected to haproxy. I don't know why this is necessary, even when dns resolver is used (there are no endpoint IP changes, it just has fixed dns record of k8s service in the backend configs).