Trying to install K8s ACS Private cluster with AKS Engine v.29 but keep on failing with errors#

[bhaskar-chenchu]

Trying to install K8s ACS Private cluster with AKS Engine v.29 but keep on failing with errors#

{ "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'cse-agent-1'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\\".\"\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'cse-agent-2'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\\".\"\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'cse-agent-0'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\\".\"\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'cse-master-1'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\\".\"\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'cse-master-2'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\\".\"\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'cse-master-0'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\\".\"\r\n }\r\n ]\r\n }\r\n}" } ] }

---

My JSON File#

{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorRelease": "1.11", "orchestratorVersion": "1.11.6", "kubernetesConfig": { "enableSecureKubelet": true, "networkPolicy": "none", "networkPlugin" :"kubenet", "clusterSubnet": "10.0.0.0/17", "serviceCidr": "10.0.128.0/17", "dnsServiceIP": "10.0.128.254", "dockerBridgeSubnet": "192.168.21.5/24", "etcdDiskSizeGB": "128", "privateCluster": { "enabled": true }, "kubeletConfig": { "--non-masquerade-cidr": "10.0.0.0/17", "--cluster-domain": "ddkdcjd"
}, "apiServerConfig": { "--enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota" }, "useInstanceMetadata": false, "enableEncryptionWithExternalKms": true } }, "masterProfile": { "count": 3, "dnsPrefix": "wmsanboxn01", "vmSize": "Standard_DS2_v2", "vnetSubnetId": "/subscriptions/xxxx/resourceGroups/we-wmsnonprod-rg/providers/Microsoft.Network/virtualNetworks/xxxxx/subnets/xxx", "firstConsecutiveStaticIP": "10.xx.xx0.xx", "vnetCidr": "10.xx.xx.0/22", "OSDiskSizeGB": 128 }, "agentPoolProfiles": [ { "name": "wmsandboxn01", "count": 3, "vmSize": "Standard_DS2_v2", "vnetSubnetId": "/subscriptions/xxx/resourceGroups/we-wmsnonprod-rg/providers/Microsoft.Network/virtualNetworks/xx/subnets/xx", "availabilityProfile": "AvailabilitySet", "storageProfile" : "ManagedDisks", "OSDiskSizeGB": 128, "acceleratedNetworkingEnabled": true } ], "linuxProfile": { "adminUsername": "admin@123si", "ssh": { "publicKeys": [ { "keyData": "ssh-rsa AAAAB3NzaC1yc2E/UIQa3kASueuSYhGk5ZkhOo9RT1ZICpMdAYoznaWvgT5V6O38dqFYKAmt3RnMHKGvwx/Vj7ay07mHVa7mdgnhGhDsRXGfQEbLGHggVHLmcjv9pukpm1wolMmz0kSf6Ohre/zYVx07DH+L+G87wehHCYbDJu/F7BmFqiF" } ] } }, "servicePrincipalProfile": { "clientId": "xxx", "secret": "xxx", "objectId": "xxx" } } }

Originally posted by @bhaskar-chenchu in https://github.com/Azure/acs-engine/issues/2595#issuecomment-457824098

[bhaskar-chenchu]

Deployment Region, West Europe.

[CecileRobertMichon]

Hi @bhaskar-chenchu, CSE exit code 50 means that we were not able to establish outbound internet connection within your cluster. See https://github.com/Azure/aks-engine/blob/master/docs/howto/troubleshooting.md#vmextensionprovisioningerror-or-vmextensionprovisioningtimeout for more help on debugging CSE errors.

Private clusters allow to block inbound traffic, not outbound. See https://github.com/Azure/aks-engine/blob/master/docs/topics/features.md#private-cluster for more details on the feature.

Please make sure that you have a valid outbound connection and that you do not have NSG rules blocking outbound internet access.

[bhaskar-chenchu]

Thanks Cecile for the response with detailed explanation.

Is there a way to specify Corporate Proxy instead of allowing outbound internet connectivity to download VM Extensions ?

[CecileRobertMichon]

Not at the moment. However, we are working on a "no outbound required" feature (still experimental). See https://github.com/Azure/aks-engine/issues/118.

Also please use aks-engine for all further issues, acs-engine is deprecated :) https://github.com/Azure/aks-engine/blob/master/docs/faq.md