[Feature] Allow configuring `serializeImagePulls` kubelet config for Windows nodes

[lippertmarkus]

Is your feature request related to a problem? Please describe. Windows images are big and especially after autoscaling when a lot of Pods scheduled on the new node simultaneously the pull time can be huge when done in serial. Example event (compare pull to wait time):

Pulled     Successfully pulled image "<image>" in 5m0343499s (22m24.5159081s including waiting)

Describe the solution you'd like I want to be able to set serializeImagePulls=false in kubelet.

Describe alternatives you've considered none

Additional context see also https://github.com/kubernetes/kubernetes/issues/108405

[artificial-aidan]

Could this be updated to allow configuring serializeImagePulls on all node types?

[lippertmarkus]

As far as I understand the docs it should be already configurable for Linux. Windows nodes are limited to only 4 configurable kubelet settings, where serializeImagePulls is not one of it.

[artificial-aidan]

As I read them serializeImagePulls isn't one of the linux settings either: https://learn.microsoft.com/en-us/azure/aks/custom-node-configuration?tabs=linux-node-pools#linux-kubelet-custom-configuration

Unless I'm misreading something.

[lippertmarkus]

Oh good catch, then we can generalize the request :)

[MikeFear]

Any updates on this? We run our e2e tests on Kubernetes on Linux nodes and therefore need to deploy a lot of microservices in a short timespan. We are currently bottlenecked by this limitation.

[allyford]

I've captured the feature request to add the serializeImagePulls parameter to custom kubelet config. We currently do not have an ETA for when this change will be made, but I'll update this issue when we do.

In the meantime, we're planning a release for #3928 for Linux this year.

[Rob19999]

Would love this one as well. I would recommend adding the maxParallelImagePulls setting as well to not overload network/disc as described in the official documentation https://kubernetes.io/blog/2023/05/15/speed-up-pod-startup/#maximum-parallel-image-pulls-will-help-secure-your-node-from-overloading-on-image-pulling.

In the mean time can we change the kubelet in a different manner then described above to test these settings?

[PlagueHO]

@allyford - I've also got teams who need this feature as well. They are currently investigating Artifact Streaming in Azure Container Registry as a stop gap, but not all images are in ACR (many on DockerHub).

[ravnalexquinn]

+1 for this! My team would also find this really useful for linux nodes

[Rob19999]

After having used this feature with a on-premise kubernetes platform and having used Artifact streaming on a AKS. I think the impact of this feature is higher then Artifact Streaming in regards to image pull performance. I noticed that you're also working on Artifact streaming for Windows. I would like that feature but exposing these parameters seems like a small change in comparison with a lot off gain.

[k-koleda]

Hey, im also interested in this feature. windows images extremely big, we have some performance gaps due to slow scaling a new node. so it would be great to have this feature available.

[gabrimonfa]

Any update on this? An ETA maybe?

Sometimes big image pulls in aks are unbearably slow, and other pulls are waiting for it to complete. Parallel pulls may help mitigate also this issue. I've opened a ticket with support with no real workarounds or solutions, apart from suggesting the use of ACR, which in certain situations is not feasible (and no, the issue does not depend on the registry, other well-known k8s managed service have no issues at all).

[artificial-aidan]

This also makes using spot nodes almost impossible. Because bringing up a new node can take longer than the spot warning is.

[pawelpabich]

Thanks!

[allyford]

AKS is planning to release this feature as default in k8s 1.31. This feature will support linux and windows. See #4499 for details and updates. Closing this feature as duplicate.