Closed R-Studio closed 8 months ago
I noticed that vpa-updater protects the newly created pod only for 1min, why?
Hey @R-Studio, thanks for providing some insight into your investigations! I'm not able to answer what's going on completely, so here's just a few pointers to clear the fog step-by-step:
Let's choose from X configs
is coming from the admission-webhook
that reacts on CREATE Pod. For each Pod, it is trying to find a matching VPA, and you currently have 5 VPAs in the Pod's namespace (kubectl -n NAMESPACE get vpa
should show this)updater
evicts a Pod in the following scenarios:
In you case, we can probably rule out cases 2 and 3, given that you're only scaling on CPU (you didn't explicitly mention this, but I was assuming it from the way you investigated this) and that the Pod was evicted just minutes before. So most likely the current requests are outside the recommended range for one of the containers in your Pod. To analyze this, it helps to also draw the upper and lower bounds and see when/how a Container's current requests end up being outside the bounds of the new recommendation (which is case 1 above). This would look like this
In this example graph, we see the "current requests" between the "lower bound", and "upper bound", so we're not in case 1. The new recommendation is lower than the current requests, though, and after 12 hours the Pod would be evicted to apply the new recommendation, as there is more than 10% difference.
A graph like this should show you, why your Pod is getting evicted and hopefully explain what happens.
@voelzmo thanks for your reply! ππ½
First you are right we are only scaling on CPU requests (sorry I forgot to mention this).
Here an example of our VPA resources:
---
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
name: vpa-argocd-applicationset-controller
namespace: argocd
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: argocd-applicationset-controller
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: '*'
minAllowed:
cpu: 10m
maxAllowed:
cpu: 3
controlledResources: ["cpu"]
controlledValues: "RequestsOnly"
Let's choose from X configs
: you are absolutely right. I thought vpa finds multiple configurations for the same pod ππ
Anyway thanks for all your inputs! ππ½
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/close /kind support
@voelzmo: Closing this issue.
/remove-kind bug
Which component are you using?: vertical-pod-autoscaler (recommender, updater & admission-controller)
What version of the component are you using?: v0.13 (Image-Tag: 0.13.0, Fairwirds-Helm Chart: v1.7.2)
Component version: What k8s version are you using (
kubectl version
)?: OpenShift 4.11, Kubernetes v1.24.12+ceaf338What environment is this in?: OnPrem VMs
What behaviour did you expect to see?: No unreasonable pod evictions/restarts
What happened instead?: Unseasonable pod evictions/restarts In the following screenshot we can see that VPA set the CPU requests from 0.055 to 0.043 for 2 minutes and then back to 0.055 again: I have noticed the following log, why 5 configs? (I only have one
VerticalPodAutoscaler
for this deployment deployed)matcher.go:68] Let's choose from 5 configs for pod NAMESPACE/xxx-85b97d6dfc-%
How to reproduce it (as minimally and precisely as possible): I am not sure if it is reproducible
I use the following arguments: Recommender:
Updater:
AdmissionController: no arguments than the defaults