Please improve the speed of retrieving the count of Service Bus Queue messages

[eugen-nw]

More exactly, this is for the "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queuemessages" metric.

As @jsturtevant confirmed in #54, the delay for the above metric to retrieve the actual count of messages in a Service Bus Message Queue can go up to 2 minutes. Since the HPA checks the count of message only every 30", it can take up to 2'30" for auto-scaling to start. The 2'30" delay is huge for our products' auto-scalability requirements, so it would be GREAT if it would be possible to make HPA's autoscaling way faster, hopefully to mere seconds over the 30" check interval.

[tomkerkhove]

Unfortunately, Azure Monitor does not report metrics that fast and checking every 30" will not make any difference, you should aim to use 1-2 min.

[eugen-nw]

HPA does the check every 30". @jsturtevant mentioned in #54 "There is a some work in progress (#42) to remove this delay go by by-passing Azure monitor and go directly to the queue directly. It also addresses a few other challenges such as scaling on queue subscription counts." so it appears that Azure Monitoring is not the only way to retrieve the count of messages in the Queue. Service Bus Explorer is accurate in showing the count of messages.

[jsturtevant]

The work for subscription count is just finishing up testing and will open up using direct service apis to get the counts instead of azure monitor. Its taken a but longer than anticipated due to other commiments but should be coming soon. Thanks for you patience.

[eugen-nw]

Thanks very much for the info! I'll wait then for this PR to get merged in https://github.com/Azure/azure-k8s-metrics-adapter/pulls Once it is merged in I'd guess that I need to reinstall and configure azure-k8s-metrics-adapter, right?

[eugen-nw]

I checked https://github.com/Azure/azure-k8s-metrics-adapter/pull/59/files and it does not appear to have addressed the counts of Messages in a Queue. Could that information be obtained directly from the Queue please?

[eugen-nw]

@jsturtevant would it be possible please to implement this sometime soon? Since we're [still] using Windows containers, this big delay further slows down our horizontal scale-out.

[byGeek]

I encounter another problem in using custom metric: request-per-second.

I follow this walkthrough: Requests per Second Custom Metric Scaling

and i deploy a webapp-test deployment using hpa.yaml.

But it seems that the metrics is delayed. I generated some load to test autoscaling, the request per second should be around 30, that supposed to scale 3~4 pods, but in aks, the replica reachs the max replica set, which is 10.

It seems that the hpa autoscaler get the SAME metric value in autoscaling. After a 2 min delay, the metric is updated, but the hpa autoscaler see un-updated metric, it keep scaling out till reach the max replica!