Arc connected clusters with empty JSON property `distribution`

[baziwane]

We experienced a regression that caused a number of Azure Arc for Kubernetes connected clusters to have the value for a JSON property distribution be set to an empty string. For AKS hybrid connected clusters, this value is usually set during initial installation and is never altered for the lifetime of the deployment.

To Reproduce Steps to reproduce the behavior:

Open an Azure PowerShell terminal and run the following commands:

1. az login
2. az account set --subscription
3. az connectedk8s show -g -n

Sample output

{
  "agentPublicKeyCertificate": <value>
  "agentVersion": "1.8.14",
  "azureHybridBenefit": "NotApplicable",
  "connectivityStatus": "Connected",
  "distribution": "",
  "distributionVersion": null,
  "id": "/subscriptions/<subscription>/resourceGroups/<resource group>/providers/Microsoft.Kubernetes/connectedClusters/<cluster name>",
  "identity": {
    "principalId": "<principal id>",
    "tenantId": "<tenant id>",
    "type": "SystemAssigned"
  },
  "infrastructure": "azure_stack_hci",
  "kubernetesVersion": "1.23.12",
  "lastConnectivityTime": "2022-11-04T14:59:59.050000+00:00",
  "location": "eastus",
  "managedIdentityCertificateExpirationTime": "2023-02-02T14:24:00+00:00",
  "miscellaneousProperties": null,
  "name": "mgmt-cluster",
  "offering": "AzureStackHCI_AKS_Management",
  "privateLinkScopeResourceId": null,
  "privateLinkState": "Disabled",
  "provisioningState": "Succeeded",
  "resourceGroup": "<resource group>",
  "systemData": {
    "createdAt": "2022-11-04T14:29:17.680060+00:00",
    "createdBy": "<>",
    "createdByType": "Application",
    "lastModifiedAt": "2022-11-04T15:00:01.830067+00:00",
    "lastModifiedBy": "64b12d6e-6549-484c-8cc6-6281839ba394",
    "lastModifiedByType": "Application"
  },
  "tags": {},
  "totalCoreCount": 4,
  "totalNodeCount": 1,
  "type": "microsoft.kubernetes/connectedclusters"
}

Expected behavior

If the output for JSON property distribution above returns an empty string, you need to follow instructions below to patch your cluster.

Fix

1. Copy the following configuration in a file called patchBody.JSON

{
  "properties": {
    "distribution": "aks_management"
  }
}

Important: If your cluster is an AKS management cluster the value should be set to aks_management. If it's a workload cluster, it should be set to aks_workload

2. From the JSON output you obtained above, copy your connected cluster's id. It should appear as a long string in the following format:

 "/subscriptions/<subscription >/resourceGroups/<resource group>/providers/Microsoft.Kubernetes/connectedClusters/<cluster name>",

3. Run the following command to patch your cluster. The <cc_arm_id> should be the Id obtained from step 2 above.

az rest -m patch -u "<cc_arm_id>?api-version=2022-10-01-preview" -b "@patchBody.json"

4. Check that your cluster has been successfully patched by running az connectedk8s show -g <rg_name> -n <cc_name> to make sure the JSON property distribution has been correctly set.

[Elektronenvolt]

@baziwane To Enable Azure Hybrid Benefits I must set aks_management at the management cluster and set the property azure-hybrid-benefit to true. Activation done.

But this article says: Workload clusters should be set to aks_workload. Is this an optional step? I did it now, but what are the consequences if I skip updating the workload clusters?

[baziwane]

@Elektronenvolt, yes workload cluster is totally optional. No consequence at the moment

[sethmanheim]

Added TSG: https://learn.microsoft.com/en-us/azure/aks/hybrid/known-issues-networking#arc-connected-clusters-have-empty-json--distribution--property