`acs-engine scale` fails: Deployment template validation failed, agent is not defined in the template

[croeck]

Is this a request for help?:

YES

---

Is this an ISSUE or FEATURE REQUEST? (choose one):

ISSUE

---

What version of acs-engine?:

Current master.

Version: canary GitCommit: 13973d0d GitTreeState: clean

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes

What happened:

Scaling the cluster fails:

acs-engine scale --subscription-id xxxxx --resource-group domain-env-kubernetes --location westeurope --deployment-dir /path_to/_output/env/ --new-node-count 2 --debug
INFO[0000] validating...
DEBU[0000] Resolving tenantID for subscriptionID: xxxxx
DEBU[0000] Attempting to load token from cache. path="yyyyy"
DEBU[0001] Saved token to cache. path="yyyyy"
DEBU[0003] Already registered for "Microsoft.Compute"
DEBU[0003] Already registered for "Microsoft.Storage"
DEBU[0003] Already registered for "Microsoft.Network"
INFO[0005] Name suffix: 87921648
INFO[0005] Starting ARM Deployment (domain-env-kubernetes-2039249985). This will take some time...
INFO[0008] Finished ARM Deployment (domain-env-kubernetes-2039249985). Error: resources.DeploymentsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidTemplate" Message="Deployment template validation failed: 'The resource 'Microsoft.Compute/virtualMachines/k8s-agent-87921648-0' is not defined in the template. Please see https://aka.ms/arm-template for usage details.'."
FATA[0008] resources.DeploymentsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidTemplate" Message="Deployment template validation failed: 'The resource 'Microsoft.Compute/virtualMachines/k8s-agent-87921648-0' is not defined in the template. Please see https://aka.ms/arm-template for usage details.'."

Previously, it was setup using the following configuration:

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.1",
      "kubernetesConfig": {
        "enableDataEncryptionAtRest": false,
        "useManagedIdentity": true,
        "enableRbac": true,
        "networkPolicy": "calico",
        "privateCluster": {
            "enabled": true
        },
        "dnsServiceIP": "10.201.96.96",
        "serviceCidr": "10.201.64.0/18",
        "clusterSubnet": "10.201.128.0/17",
        "etcdDiskSizeGB": "256",
        "dockerEngineVersion": "17.05.*"
      }
    },
    "aadProfile": {
      "serverAppID": "******",
      "clientAppID": "******",
      "tenantID": "******",
      "adminGroupID": "******"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "env",
      "vmSize": "Standard_A4_v2",
      "vnetSubnetId": "/subscriptions/******/resourceGroups/domain-env-kubernetes/providers/Microsoft.Network/virtualNetworks/env-k8s-network/subnets/env-k8s-master-subnet",
      "firstConsecutiveStaticIP": "10.201.24.10",
      "vnetCidr": "10.201.0.0/16"
    },
    "agentPoolProfiles": [
      {
        "name": "agent",
        "count": 1,
        "vmSize": "Standard_A4_v2",
        "vnetSubnetId": "/subscriptions/******/resourceGroups/domain-env-kubernetes/providers/Microsoft.Network/virtualNetworks/env-k8s-network/subnets/env-k8s-worker-subnet",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "******",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "******"
          }
        ]
      }
    }
  }
}

We used these commands to deploy the cluster:

acs-engine generate template.json
az group deployment validate --resource-group domain-env-kubernetes --template-file /path_to/_output/env/azuredeploy.json --parameters /path_to/_output/env/azuredeploy.parameters.json
az group deployment create --name domain-env-k8s-cluster --resource-group domain-env-kubernetes --template-file /path_to/_output/env/azuredeploy.json --parameters /path_to/_output/env/azuredeploy.parameters.json

The rollout was successful and the cluster fully functional.

What you expected to happen:

Cluster scaled to 2 nodes.

How to reproduce it (as minimally and precisely as possible):

As described in what happend

Anything else we need to know:

• Custom VNET
• Tested with multiple master and multiple node setup as well, same result.

[CecileRobertMichon]

@jackfrancis this is the scale bug I mentioned earlier that @danigian reported.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.