seguler
commented
1 year ago Thanks for sharing this.
What is your application SLO? How does the freeze events impact your application SLO? Could you elaborate on what impact you're observing?
We have a feature called node autodrain that can evict applications for certain scheduled events but we do not currently evict for freeze events. Freeze events for the most part don't have any impact on the applications. In some cases, you may see a few seconds of impact (VM freeze, IO latency, some percent TCP drop).
In my opinion, such failures are very normal and expected per the publicly documented VM SLAs. We all should build our applications to be resilient to them (e.g. user TCP timeouts, automatic retries etc.).
We do plan to provide an option for customers to choose to automatically evict pods during a freeze event. However, in our internal tests, we actually observed more impact to our application SLO when we enabled to evict during freeze events. Freeze events are frequent, and they hit every node eventually. Evicting and running away from it did not reduce impact in our case. Impact was multitude greater when we tried to run away from them. Fwiw, we rarely saw an impact from freeze events due to all retries and resiliency mechanisms in our applications.
I do like the idea you have that perhaps we should update readiness probes during the freeze so that pods don't receive traffic. However, if the node was in bad shape, load balancer would probe it down after 10 seconds anyway. Is the impact you're not happy about just less than 10 seconds?
dtotopus
ashwajce
grzesuav
Azure VM Instabilities and Scheduled Events Impacting Service Stability
Issue Description
We are experiencing instabilities with AKS because of Azure virtual machines (VMs), specifically related to scheduled events causing VM freezes. These instabilities are impacting our service stability and operational efficiency as we get some of our nodes frozen for several seconds few times a week.
Source of Information
We have referred to the official Azure documentation on scheduled events[https://learn.microsoft.com/en-us/azure/virtual-machines/linux/scheduled-events], available at Azure VM Scheduled Events, for a better understanding of the issue.
Expected Behavior
Scheduled events are intended to provide advance notice of underlying maintenance tasks or events impacting Azure VMs. These events should be handled seamlessly by the Azure infrastructure, ensuring minimal or no service disruption for the customers. We expect the VMs to be able to handle scheduled events without freezing or negatively impacting service availability.
Current Behavior
Unfortunately, our experience with AKS with scheduled events has been far from satisfactory. The following issues have been observed:
VM Freezes
When a scheduled event is triggered events like Azure VMs freeze may cause partial or complete service outage. As by default such events are not handled by AKS - it will detect frozen nodes but will not evict pods out of it. As a result some of the traffic will be served to the frozen endpoints. This can be addressed by tracking vm events using Azure api but this doesn't make it any better as freezes are usually pretty short 6-30 seconds - meaning that any operation like draining the node will take even more time and cause more chaos. It would also affect workloads that can not be stopped immediatelly so we can't really drain the node in advance in all cases. Basically this maintenance "feature" renders regular instances as unstable as spot ones.
Inadequate Notification Time
The minimal notification period provided ( 15 minutes ) prior to the scheduled event is insufficient to take necessary preventive measures. This short notice period severely limits ability to prepare and mitigate any potential impact on our services as we have some services that do certain tasks which need more than 30 minutes of stable uptime. In contrary AWS sends an email days/weeks before maintenance event ( which are very rare as well ) and in some cases allows you to change the event date.
Lack of Flexible Control
We are unable to control or influence the timing of scheduled events. As a result, these events may occur during peak business hours or critical operational periods, exacerbating the impact on our services. It is possible to develop a scheduler that would automatically handle failover in case of such events but such approach feels really weird as in case of AKS - we would be writing a node scheduler on top of Kubernetes which is already an orchestrator. And as before - this wouldn't solve stability requirement for long running jobs or databases.
Impact
The current state of Azure VM freezes and AKS inabillity to track scheduled events affects our service stability, availability, and customer satisfaction. The ongoing VM freezes during scheduled events, along with the short notification time, pose challenges and hinder our operational efficiency. It is essential for us to have a reliable and predictable VM environment to ensure the continuous delivery of our services.
Expected Solution
Improved Handling of Scheduled Events
We do expect either AKS to be able to handle scheduled freezes on it's own by rendering nodes visibly unavailable and temporarely stop sending traffic to them. Or to develop a mechanism that ensures Azure VMs can handle scheduled events without freezing or causing service interruptions. It is crucial that these events are processed seamlessly in the background, preventing any negative impact on our services.
Extended Notification Time
We request an increase in the notification time provided for scheduled events. A minimum of few day prior notification would significantly help us in preparing for any upcoming maintenance tasks or events and reducing service disruptions.
Additional Information
Providing the log of VM freezes events we've experienced in a week