[BUG] azure-vnet-telemetry leaks sockets

[geertn]

Describe the bug The process azure-vnet-telemetry seems to be leaking sockets. Errors are also present in the logs. Every error line corresponds to a socket leak. They happen on multiple nodes in the AKS cluster but not all. The issue has been present for weeks and caused a production outage because of too many open sockets.

To Reproduce

• See if the node produces errors like this:

  2024/04/10 15:41:16 [4791] StartServer: unmarshal error:invalid character 'e' looking for beginning of value
  2024/04/10 15:42:12 [4791] StartServer: unmarshal error:invalid character 'r' looking for beginning of value

• Look at open sockets:

  root@aks-systemz1-13451524-vmss000000:/# netstat -anp | grep telemetr | wc -l
  43
  root@aks-systemz1-13451524-vmss000000:/# 

• Most of the times the error and amount of open sockets correspond

  root@aks-userz2-26395965-vmss00000Q:/# netstat -anp | grep telemetr | wc -l
  337
  root@aks-userz2-26395965-vmss00000Q:/# cat /var/log/azure-vnet-telemetry.log  | grep "unmarshal error" | wc -l
  336
  root@aks-userz2-26395965-vmss00000Q:/# 

Expected behavior No errors and no leaking sockets.

Environment (please complete the following information):

• Kubernetes version 1.27.9, latest node image

k get nodes -o wide
NAME                               STATUS   ROLES   AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-systemz1-13451524-vmss000000   Ready    agent   4h1m    v1.27.9   10.106.16.4     <none>        Ubuntu 22.04.4 LTS   5.15.0-1059-azure   containerd://1.7.14-1

Additional context

logfile.txt

Open support issue 2403170050000665

[tamilmani1989]

Thanks for letting know.. We are tracking the issue here https://github.com/Azure/azure-container-networking/issues/2703 and will be working on a fix. I will keep this thread updated

[geertn]

I see the fix is merged, thank you for that. Will this be included in the upcoming node image release?