search

Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.97k stars 308 forks source link

[Feature] Artifact Stream create too much error logs #4194

Closed nowjean closed 7 months ago

nowjean commented 7 months ago

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I Enabled Artifact Streaming on a new node pool, and applied create streaming artifact to one of my acr image. In this situations, when i delete artifact streaming the acr image, i got a lot of Evicted pods on the node pool.

When i describe the node, i found a disk pressure error. Disk pressure made by a huge log files in /var/log/messages and syslog on the node.

Belows are the error what i got.

Apr  1 10:09:29 aks-nodepool-28456089-vmss00004O acr[4090]: #033[2m2024-04-01T10:09:28.578384Z#033[0m #033[33m WARN#033[0m #033[2mlifec_registry::plugins::kube::watcher::pod_index#033[0m#033[2m:#033[0m Received empty event
Apr  1 10:09:29 aks-nodepool-28456089-vmss00004O acr[4090]: #033[2m2024-04-01T10:09:28.578387Z#033[0m #033[33m

-rw-r----- 1 syslog adm 61954191360 Apr 4 09:15 syslog -rw-r----- 1 syslog adm 61954215936 Apr 4 09:15 messages

Describe the solution you'd like I think it should be create alert without error log or reduce amount of error.

juliusl commented 7 months ago

@nowjean Hello thanks for reporting this issue. We have a fix for this that is currently being released.

nowjean commented 7 months ago

@juliusl Happy to hear that! Thanks!

ganeshkumarashok commented 7 months ago

Fix is in the 202403.13.0 VHD and it's being rolled out as Julius mentioned. You can track it here: https://releases.aks.azure.com/webpage/index.html

ganeshkumarashok commented 7 months ago

The fix is already in most regions

philwelz commented 7 months ago

@nowjean i will close this issue as the answer was provided by Julius and Ganesh. feel free to re-open.