AKS Load Balancer - cannot use public IP address

[Puiu]

Problem description

I am trying to create a load balancer for a Kubernetes deployment with an azure public IP address, but I get this error:

Errors & Logs

kubernetes:core:Service (anabi-api-service): error: 2 errors occurred:

• resource default/anabi-api-service-1jk62ox7 was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'anabi-api-service-1jk62ox7' timed out waiting to be Ready
• Service was not allocated an IP address; does your cloud provider support this?

Reproducing the issue

My code looks like this:

.....

const apiLoadBalancerIp = new azure.network.PublicIp("LBIp", {
         allocationMethod: "Static",
         resourceGroupName: resourceGroup.name,
         tags: {
             service: "kubernetes-api-loadbalancer",
         },
     });

const apiLoadBalancerFirewallRule = new azure.sql.FirewallRule("api-lb", {
        endIpAddress: apiLoadBalancerIp.ipAddress,
        resourceGroupName: resourceGroup.name,
        serverName: sqlServer.name,
        startIpAddress: apiLoadBalancerIp.ipAddress,
    });

const anabiapiService = new k8s.core.v1.Service("anabi-api-service", {
    metadata: {
        labels: anabiapiDeployment.spec.template.metadata.labels
    },
    spec: {
        type: "LoadBalancer",
        ports: [{
            port: 80,
            targetPort: 3000
        }],
        loadBalancerIP: apiLoadBalancerIp.ipAddress,
        selector: anabiapiDeployment.spec.template.metadata.labels
    }
}, { provider: aksprovider, dependsOn: [apiLoadBalancerFirewallRule]});

I know you can get the ip of the load balancer after it was created, and it works. But trying to do it with pre-existing IP, it fails.

[mikhailshilkov]

I faced the same problem today, which I managed to solve, so I'm dumping my solution here to help anybody else.

After the failed deproyment of the service, I ran kubectl get services:

NAME               TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
app-svc-duevx4ka   LoadBalancer   10.2.2.134   <pending>     80:32263/TCP   2m35s

I see that external IP is indeed missing. I then ran kubectl describe service app-svc-duevx4ka to get the actual error:

user supplied IP Address 52.174.253.207 was not found in resource group 
mc_fsakse1ec8b14_fsakscluster22b4d410_westeurope

It turns out this is because my Public IP was in my own resource group, but it has to be in the resources group where the cluster nodes are provisioned.

@Puiu In your code above you should change the Public IP definition to

const apiLoadBalancerIp = new azure.network.PublicIp("LBIp", {
         allocationMethod: "Static",
         resourceGroupName: cluster.nodeResourceGroup,
         tags: {
             service: "kubernetes-api-loadbalancer",
         },
     });

a full example is available here.

Another possible error that I also got later is

LinkedAuthorizationFailed\" Message=\"The client 'guid' with object id 'guid' has 
permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/virtualmachines/write' 
on scope '/subscriptions/guid/resourceGroups/mc_fsakse1ec8b14_fsakscluster21b4d410_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-aksagentpool-33998035-vmss/virtualmachines/2'; 
however, it does not have permission to perform action 'Microsoft.Network/virtualNetworks/subnets/join/action' 
on the linked scope(s) '/subscriptions/guid/resourceGroups/fsakse1ec8b14/providers/Microsoft.Network/virtualNetworks/fsaksvnetd0e531c8/subnets/fsakssubnetd21d113f' 
or the linked scope(s) are invalid.

which means that the service principal doesn't have enough permissions to deploy the networking changes. Lack of permission would err the deployment of any load balancer, with or without an explicit IP.

Be sure to assign your SP to the Network Contributor role:

const rgNetworkRole = new azure.role.Assignment("assignment", {
    principalId: servicePrincipal.id,
    scope: resourceGroup.id,
    roleDefinitionName: "Network Contributor"
});

This fixed the error only after I redeployed the whole AKS cluster.

[lblackstone]

It looks like https://github.com/pulumi/pulumi-kubernetes/issues/945#issuecomment-586295133 provides a workaround, so closing this out.

[diavrank]

Hi, I'm trying to deploy an AKS cluster from this repo https://github.com/pulumi/examples/tree/master/kubernetes-ts-multicloud but I'm getting this error:

 kubernetes:core/v1:Service (aks-demo-app):
    error: 2 errors occurred:
        * the Kubernetes API server reported that "default/aks-demo-app-0d237785" failed to fully initialize or become live: 'aks-demo-app-0d237785' timed out waiting to be Ready
        * Service was not allocated an IP address; does your cloud provider support this?

I already modified some parts of the code which were deprecated to solve other issues but I'm still getting the above error:

In my service appears like:

NAME                    TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
aks-demo-app-1df19714   LoadBalancer   10.2.2.18    <pending>     80:30560/TCP   17m

So, I described it and this is the cause:

 Type     Reason                  Age                  From                Message
  ----     ------                  ----                 ----                -------
  Normal   EnsuringLoadBalancer    4m53s (x8 over 15m)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  4m52s (x8 over 15m)  service-controller  Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {\r
  "error": {\r
    "code": "PublicIPAndLBSkuDoNotMatch",\r
    "message": "Standard sku load balancer /subscriptions/dcc0660f-f795-4497-9301-022f92ct333s/resourceGroups/mc_multicloud89299de2_aksclusterbd3b44d_westus2/providers/Microsoft.Network/loadBalancers/kubernetes cannot reference Basic sku publicIP /subscriptions/dcc0660f-f795-4497-9301-022f92ct333s/resourceGroups/MC_multicloud89299de2_aksClusterbd3b44d_westus2/providers/Microsoft.Network/publicIPAddresses/staticappipa6e0d4e5.",\r
    "details": []\r
  }\r
}

Maybe this is the cause of the problem https://github.com/MicrosoftDocs/azure-docs/issues/44148#issuecomment-562313602 , but I'm not sure how to modify my pulumi program to fix it.

Any help??

Thanks in advance

[diavrank]

I just found that we can change it through the sku property:

so, the code should be updated as shown:

this.staticAppIP = new azure.network.PublicIp("staticAppIP", {
            resourceGroupName: this.cluster.nodeResourceGroup,
            allocationMethod: "Static",
            sku: "Standard", // By default, standard load balancer is used when you create a new cluster instead of basic
            location: resourceGroup.location,
        }, {parent: this}).ipAddress;

With that, I was able to see the web app exposed to the internet successfully!!!