Any tips for performance tuning?

[vitobotta]

Hi, and merry Christmas!

I was wondering if there are any tips or best practices with regards to haproxy ingress and performance? Thanks!

[jcmoraisjr]

Hi, yes, some guidelines below.

If you want to scale a high load:

• use at least v0.8 - older versions cannot even be compared.
• use dedicated hosts with at least one core per thread + one core to the ingress controller + one extra core if your cluster is big (about 3000 ingress or more) and noisy (the cluster is changing all the time, which leads to the controller frequently parsing ingress objects). Note that it's a common practice to use the workload nodes as ingress controller nodes, this is not a problem provided that your workload leaves some cores to the proxy.
• tune nbthread config key - our cluster doesn't scale with more than 3 threads despite the number of cores, and 3 is about 2x better than 1
• use dynamic config (config key) and reusesocket as the reload strategy (command-line) which will save, respectively, a lot of cpu and some random connection failures from clients. Btw they are the default since v0.8.

Another tips to have less latency in the proxy despite the load of your cluster:

• use v0.9 if you need client cert authentication and don't use ssl-passthrough - you'll avoid a boring tcp proxy in front of the main frontend which leads to latency
• do use only global client timeout config - do not use client timeout per ingress/hostname - you will avoid to split the frontend into two or more and create the boring tcp proxy which leads to latency
• avoid ssl-passthrough if possible, due to the boring tcp proxy which need to be created for such configuration

There are also some planned improvements in future versions - the current code still generates a somewhat noisy frontend, depending of your configuration, which sometimes create some latency. Keep your controller always up to date - both haproxy and the configuration generated by haproxy-ingress will continue to improve performance specially in some corner cases.

Merry Christmas =)

[vitobotta]

Hi @jcmoraisjr ! Thanks a lot for these tips, I will study them and try to apply.

I'm having a problem though with haproxy installed in a brand new cluster. It seems that the controller tries to reload/restart the backend in a loop with errors like this:

haproxy-ingress-controller-jctk6 haproxy-ingress E1224 23:56:05.257472       7 controller.go:337] unexpected failure restarting the backend:
haproxy-ingress-controller-jctk6 haproxy-ingress exit status 1
haproxy-ingress-controller-jctk6 haproxy-ingress W1224 23:56:05.259159       7 queue.go:113] requeuing kube-system/cloud-controller-manager-token-hzc5k, err exit status 1
haproxy-ingress-controller-jctk6 haproxy-ingress I1224 23:56:07.227931       7 controller.go:332] backend reload required
haproxy-ingress-controller-jctk6 haproxy-ingress I1224 23:56:07.258327       7 controller.go:209] HAProxy[pid=236] output:
haproxy-ingress-controller-jctk6 haproxy-ingress 2019/12/24 23:56:07 socat[239] E connect(5, AF=1 "/var/run/haproxy-stats.sock", 29): Connection refused
haproxy-ingress-controller-jctk6 haproxy-ingress E1224 23:56:07.258373       7 controller.go:337] unexpected failure restarting the backend:
haproxy-ingress-controller-jctk6 haproxy-ingress exit status 1
haproxy-ingress-controller-jctk6 haproxy-ingress W1224 23:56:07.261352       7 queue.go:113] requeuing fip-controller/fip-controller-token-gp64r, err exit status 1
haproxy-ingress-controller-jctk6 haproxy-ingress I1224 23:56:09.227958       7 controller.go:332] backend reload required
haproxy-ingress-controller-jctk6 haproxy-ingress I1224 23:56:09.257322       7 controller.go:209] HAProxy[pid=241] output:
haproxy-ingress-controller-jctk6 haproxy-ingress 2019/12/24 23:56:09 socat[244] E connect(5, AF=1 "/var/run/haproxy-stats.sock", 29): Connection refused
haproxy-ingress-controller-jctk6 haproxy-ingress E1224 23:56:09.257354       7 controller.go:337] unexpected failure restarting the backend:
haproxy-ingress-controller-jctk6 haproxy-ingress exit status 1
haproxy-ingress-controller-jctk6 haproxy-ingress W1224 23:56:09.257365       7 queue.go:113] requeuing cattle-system/default-token-jnvnl, err exit status 1

There is that connection refused and what is it doing with those tokens?

Thanks!

[vitobotta]

The ingresses seem to work though. It's just that there is a lot of noise in the logs and it seems to restart all the time

[vitobotta]

Interesting, setting the tag to 0.8 from the default 0.7.2 seems to have fixed it, no more noise in the logs :)