Closed JRBANCEL closed 4 years ago
Since it's my doing, I'll try to fix it. /assign
@vagababov, the code used to generate the load: https://github.com/JRBANCEL/Experimental/blob/master/KnativeBenchmarking/cmd/vegeta/main.go
The latest changes should have dealt with it. @JRBANCEL can you verify you no longer see this when running Knative from HEAD?
/close
@vagababov: Closing this issue.
What version of Knative?
v0.12.0-86-gbb1415ea8-dirty
No ConfigMap was tweaked.Expected Behavior
I am working on reaching 1M QPS. I have a simple Vegeta code that sends requests to the hello world example. Every 30s, the QPS increase by 100. Every 1k increase, the QPS is maintained for 2 minutes to evaluate stability.
I am expecting the
ksvc
to gradually scale, one pod at a time and the latency to stay low. In this case, I am not even expecting scaling since we are talking about only a few thousands QPS.Actual Behavior
Randomly, Autoscaler goes into panic mode and scales the revision for no good reason and then unpanics and goes back to where it was before. QPS Dashboard Autscaler Dashboard
With @vagababov, we found out in the logs that
ObservedStableValue
goes up ~2x before the panic, then comes back to value it was before. @vagababov suggested it could be double counting because of the bucketing.