Can't create services with internal load balancer - failed to ensure host in pool: "azure - ARM rate limited(read) for operation:NicGet"

[yfried]

Is this a request for help?:

YES

Is this an ISSUE or FEATURE REQUEST? (choose one):

ISSUE

What version of acs-engine?:

• v0.26.0

• v0.25.3

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes 1.11.4

What happened: Internal load balancers stuck in pending with the following error:

failed to ensure host in pool: "azure - ARM rate limited(read) for operation:NicGet", ensure(default/ingress-dns-nginx-ingress-controller): backendPoolID(/LB_ID) - failed to ensure host in pool: "azure - ARM rate limited(read) for operation:NicGet", ensure(default/ingress-dns-nginx-ingress-controller): backendPoolID(/LB_ID) - failed to ensure host in pool: "azure - ARM rate limited(read) for operation:NicGet", ensure(default/ingress-dns-nginx-ingress-controller): backendPoolID(/LB_ID) - failed to ensure host in pool: "azure - ARM rate limited(read) for operation:NicGet"

What you expected to happen: Internal lb to be created in resource group and assigned to service

How to reproduce it (as minimally and precisely as possible):

helm install stable/nginx-ingress  --name ingress-dns --set service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal=true

Anything else we need to know: I'm using

{
  "kubernetesConfig":
  {
    "networkPlugin": "kubenet",
    "networkPolicy": "calico",
    "privateCluster": {"enabled": true}
  }
}

Trying to set "cloudProviderRateLimit": false hits #4143

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.