Error when upgrading or scaling an AKS cluster

[larslevie]

When trying to az upgrade or az scale a AKS cluster I just created from scratch, I get the following error:

Operation failed with status: 'Bad Request'. Details: Changing property 'linuxProfile.ssh.publicKeys.keyData' is not allowed.

Steps to reproduce:

1. Make sure container service is registered

>> az provider show -n Microsoft.ContainerService
Namespace                   RegistrationState
--------------------------  -------------------
Microsoft.ContainerService  Registered

2. Create resource group

>> az group create -l centralus -n am26-kube
Location    Name
----------  ---------
centralus   am26-kube

3. Create new cluster

   >> az aks create -n am26-labstack -g am26-kube --ssh-key-value ~/.ssh/labstack_rsa.pub
   Location    Name           ResourceGroup
   ----------  -------------  ---------------
   centralus   am26-labstack  am26-kube

4. Check cluster version:

>> az aks get-versions -n am26-labstack -g am26-kube
Name     ResourceGroup    MasterVersion    MasterUpgrades       NodePoolVersion    NodePoolUpgrades
-------  ---------------  ---------------  -------------------  -----------------  -------------------
default  am26-kube        1.7.7            1.8.2, 1.7.9, 1.8.1  1.7.7              1.8.2, 1.7.9, 1.8.1

4. Try to upgrade to available version

>> az aks upgrade -n am26-labstack -g am26-kube -k 1.8.2
Kubernetes may be unavailable during cluster upgrades.
Are you sure you want to perform this operation? (y/n): y
Operation failed with status: 'Bad Request'. Details: Changing property 'linuxProfile.ssh.publicKeys.keyData' is not allowed.

5. Try to scale the cluster

>> az aks scale -n am26-labstack -g am26-kube -c 1
Operation failed with status: 'Bad Request'. Details: Changing property 'linuxProfile.ssh.publicKeys.keyData' is not allowed.

[amanohar]

@mboersma do you know how upgrade/create sets SSH key? In this case SSH key was located at ~/.ssh/labstack_rsa.pub during create

@larslevie to confirm the SSH key at ~/.ssh/labstack_rsa.pub did not change between when create was issue and when upgrade was issued?

[larslevie]

That's correct, that key was created before the steps described above, and has not changed.

[larslevie]

I just created and upgraded a test cluster successfully by allowing Azure to generate the SSH keys for me.

>> az aks create -n am26-labstack -g am26-kube --generate-ssh-keys
Location    Name           ResourceGroup
----------  -------------  ---------------
centralus   am26-labstack  am26-kube

az aks get-versions -n am26-labstack -g am26-kube
Name     ResourceGroup    MasterVersion    MasterUpgrades       NodePoolVersion    NodePoolUpgrades
-------  ---------------  ---------------  -------------------  -----------------  -------------------
default  am26-kube        1.7.7            1.8.2, 1.7.9, 1.8.1  1.7.7              1.8.2, 1.7.9, 1.8.1

az aks upgrade -n am26-labstack -g am26-kube -k 1.8.2
Kubernetes may be unavailable during cluster upgrades.
Are you sure you want to perform this operation? (y/n): y
Location    Name           ResourceGroup
----------  -------------  ---------------
centralus   am26-labstack  am26-kube

[mboersma]

@amanohar the CLI GETs the managed cluster's JSON representation, modifies the kubernetes_version field, and PUTs it back to the server for az aks upgrade. So it seems unlikely the SSH keyData field could be changed by the CLI itself after the initial az aks create.

I haven't been able to reproduce this by following similar steps (but in the eastus region):

$ az --version | head -n 1
azure-cli (2.0.22)
$ az group create -l eastus -n maboersm
$ az aks create -g maboersm -n maboersm02 --ssh-key-value=~/.ssh/id_sprout_rsa.pub
$ az aks upgrade -g maboersm -n maboersm02 -k 1.8.2  # --debug
$ az aks scale -g maboersm -n maboersm02 -c 5

No errors, cluster looks good. @larslevie are you using the current 2.0.22 az CLI, and is this still reproducible for you?

[larslevie]

Yes, I can still reproduce if I bring my own SSH key. I'm running azure-cli 2.0.21, which it looks like is the latest stable release.

[mboersma]

@larslevie I've tried to reproduce this several times now, using the 2.0.21 CLI explicitly and various RSA keys I had and ones I generated (with and without passwords), but I can't make this error happen.

If anyone has seen this or has an idea how to reproduce it, please let me know.

[slack]

Going to close this issue out, please open a new issue if you continue to have problems.