Azure / azure-cli

Azure Command-Line Interface
MIT License
4.03k stars 3.01k forks source link

Attempting to update AKS cluster outbound type to managedNATGateway returns invalid value #27990

Closed xec-abailey closed 7 months ago

xec-abailey commented 11 months ago

Describe the bug

Attempting to update a cluster (Kubernetes 1.28.3) to use managedNATGateway rather than default loadBalancer via the command:

az aks update -n cluster -g group --outbound-type managedNATGateway --nat-gateway-managed-outbound-ip-count 16

Which returns:

Invalid outbound type, supported values are loadBalancer, managedNATGateway, userAssignedNATGateway and userDefinedRouting.

Unsure as to the cause of this since the outbound type name is clearly the same.

Related command

az aks update -n cluster -g group --outbound-type managedNATGateway --nat-gateway-managed-outbound-ip-count 16

Errors

Invalid outbound type, supported values are loadBalancer, managedNATGateway, userAssignedNATGateway and userDefinedRouting.`

Issue script & Debug output

cli.knack.cli: Command arguments: ['aks', 'update',  [redacted],  '--outbound-type', 'managedNATGateway', '--nat-gateway-managed-outbound-ip-count', '16', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fc60dc83060>, <function OutputProducer.on_global_arguments at 0x7fc60db9e520>, <function CLIQuery.on_global_arguments at 0x7fc60db782c0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'aks': ['azure.cli.command_modules.acs']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: acs                       0.117         7        54
cli.azure.cli.core: Total (1)                 0.117         7        54
cli.azure.cli.core: Loaded 7 groups, 54 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : aks update
cli.azure.cli.core: Command table: aks update
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fc60cf067a0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2023-12-08.15-54-17.aks_update.12317.log'.
az_command_data_logger: command args: aks update -n {} -g {} --outbound-type {} --nat-gateway-managed-outbound-ip-count {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7fc60cde0ae0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7fc60cde2ca0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7fc60cde2de0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fc60db9e5c0>, <function CLIQuery.handle_query_parameter at 0x7fc60db78360>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fc60cde2d40>]
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ContainerServiceClient
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /root/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
en', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/ [redacted]/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/6c10', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsofe', 'device_authorization_endpoint': 'https://login.microsoftonline.com/e', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com//kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net/ [redacted]/.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net/ [redacted]/.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: a377268f-20de-4459-869a-1e674e3b9870
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/ [redacted]/providers/Microsoft.ContainerService/managedClusters/kc-dev?api-version=2023-10-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '0ac50770-95e2-11ee-bcf3-002248b3c996'
cli.azure.cli.core.sdk.policies:     'CommandName': 'aks update'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '-n -g --outbound-type --nat-gateway-managed-outbound-ip-count --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.55.0 (PIP) azsdk-python-azure-mgmt-containerservice/28.0.0 Python/3.11.7 (Linux-6.2.0-1016-azure-x86_64-with-glibc2.36)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/[redacted]" 200 None
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Transfer-Encoding': 'chunked'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Content-Encoding': 'gzip'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'Vary': 'Accept-Encoding'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'WESTUS2:20231208T155418Z:29e4ad3b-4b4a-45d2-bbd0-a3e742b9db11'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '11996'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '29e4ad3b-4b4a-45d2-bbd0-a3e742b9db11'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': 'd2fa0971-8a0f-49a9-a7cf-cdd04922c45f'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'Date': 'Fri, 08 Dec 2023 15:54:17 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
  "id": "/subscriptions/[redacted]/providers/Microsoft.ContainerService/managedClusters/kc-dev",
  "location": "centralus",
  "name": "kc-dev",
  "tags": {
   "Environment": "DEV"
  },
  "type": "Microsoft.ContainerService/ManagedClusters",
  "properties": {
   "provisioningState": "Succeeded",
   "powerState": {
    "code": "Running"
   },
   "kubernetesVersion": "1.28.3",
   "currentKubernetesVersion": "1.28.3",
   "dnsPrefix": "kc-dev-dns",
   "fqdn": "kc-dev-dns-3748b7b4.hcp.centralus.azmk8s.io",
   "azurePortalFQDN": "kc-dev-dns-3748b7b4.portal.hcp.centralus.azmk8s.io",
   "agentPoolProfiles": [
    {
     "name": "f64v2",
     "count": 0,
     "vmSize": "Standard_F64s_v2",
     "osDiskSizeGB": 128,
     "osDiskType": "Ephemeral",
     "kubeletDiskType": "OS",
     "vnetSubnetID": "/subscriptions/[redacted]/providers/Microsoft.Network/virtualNetworks/rg-dev-k8-vnet/subnets/default",
     "maxPods": 30,
     "type": "VirtualMachineScaleSets",
     "maxCount": 20,
     "minCount": 0,
     "enableAutoScaling": true,
     "provisioningState": "Succeeded",
     "powerState": {
      "code": "Running"
     },
     "orchestratorVersion": "1.28.3",
     "currentOrchestratorVersion": "1.28.3",
     "enableNodePublicIP": false,
     "nodeTaints": [
      "HPC=True:NoSchedule"
     ],
     "mode": "User",
     "osType": "Linux",
     "osSKU": "Ubuntu",
     "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202310.31.0",
     "upgradeSettings": {},
     "enableFIPS": false
    },
    {
     "name": "system",
     "count": 2,
     "vmSize": "Standard_D8ds_v5",
     "osDiskSizeGB": 128,
     "osDiskType": "Ephemeral",
     "kubeletDiskType": "OS",
     "vnetSubnetID": "/subscriptions/[redacted]providers/Microsoft.Network/virtualNetworks/rg-dev-k8-vnet/subnets/default",
     "maxPods": 30,
     "type": "VirtualMachineScaleSets",
     "maxCount": 20,
     "minCount": 1,
     "enableAutoScaling": true,
     "scaleDownMode": "Delete",
     "provisioningState": "Succeeded",
     "powerState": {
      "code": "Running"
     },
     "orchestratorVersion": "1.28.3",
     "currentOrchestratorVersion": "1.28.3",
     "enableNodePublicIP": false,
     "mode": "System",
     "enableEncryptionAtHost": false,
     "enableUltraSSD": false,
     "osType": "Linux",
     "osSKU": "Ubuntu",
     "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202310.31.0",
     "upgradeSettings": {},
     "enableFIPS": false
    }
   ],
   "windowsProfile": {
    "adminUsername": "azureuser",
    "enableCSIProxy": true
   },
   "servicePrincipalProfile": {
    "clientId": "msi"
   },
   "addonProfiles": {
    "aciConnectorLinux": {
     "enabled": true,
     "config": {
      "SubnetName": "virtual-node-aci"
     },
     "identity": {
      "resourceId": "/subscriptions/[redacted]/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aciconnectorlinux-kc-dev",
      "clientId": [redacted],
      "objectId":  [redacted]
     }
    },
    "azurepolicy": {
     "enabled": true,
     "config": null,
     "identity": {
      "resourceId": "/subscriptions/[redacted]/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-kc-dev",
      "clientId":  [redacted],
      "objectId": [redacted]
     }
    },
    "httpApplicationRouting": {
     "enabled": false,
     "config": {
      "HTTPApplicationRoutingZoneName": "8509409c7522409a8a75.centralus.aksapp.io"
     }
    },
    "ingressApplicationGateway": {
     "enabled": false,
     "config": null
    },
    "omsAgent": {
     "enabled": true,
     "config": {
      "logAnalyticsWorkspaceResourceID": "/subscriptions/[redacted]/providers/Microsoft.OperationalInsights/workspaces/ [redacted]
     },
     "identity": {
      "resourceId": "/subscriptions/[redacted]/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-kc-dev",
      "clientId":[redacted],
      "objectId": [redacted]
     }
    }
   },
   "nodeResourceGroup": [redacted],
   "enableRBAC": true,
   "supportPlan": "KubernetesOfficial",
   "networkProfile": {
    "networkPlugin": "azure",
    "networkDataplane": "azure",
    "loadBalancerSku": "Standard",
    "loadBalancerProfile": {
     "managedOutboundIPs": {
      "count": 1
     },
     "effectiveOutboundIPs": [
      {
       "id": "/subscriptions/ [redacted]/providers/Microsoft.Network/publicIPAddresses/63ed83a4-5aef-41b8-bf59-802a4b4027ad"
      }
     ]
    },
    "serviceCidr": "10.0.0.0/16",
    "dnsServiceIP": "10.0.0.10",
    "outboundType": "loadBalancer"
   },
   "maxAgentPools": 100,
   "identityProfile": {
    "kubeletidentity": {
     "resourceId": " [redacted]/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kc-dev-agentpool",
     "clientId": [redacted],
     "objectId":  [redacted]
    }
   },
   "autoScalerProfile": {
    "balance-similar-node-groups": "false",
    "expander": "random",
    "max-empty-bulk-delete": "10",
    "max-graceful-termination-sec": "600",
    "max-node-provision-time": "15m",
    "max-total-unready-percentage": "45",
    "new-pod-scale-up-delay": "0s",
    "ok-total-unready-count": "3",
    "scale-down-delay-after-add": "10m",
    "scale-down-delay-after-delete": "10s",
    "scale-down-delay-after-failure": "3m",
    "scale-down-unneeded-time": "10m",
    "scale-down-unready-time": "20m",
    "scale-down-utilization-threshold": "0.5",
    "scan-interval": "10s",
    "skip-nodes-with-local-storage": "false",
    "skip-nodes-with-system-pods": "true"
   },
   "autoUpgradeProfile": {
    "upgradeChannel": "rapid"
   },
   "securityProfile": {},
   "storageProfile": {
    "diskCSIDriver": {
     "enabled": true
    },
    "fileCSIDriver": {
     "enabled": true
    },
    "snapshotController": {
     "enabled": true
    }
   },
   "workloadAutoScalerProfile": {},
   "azureMonitorProfile": {
    "metrics": {
     "enabled": true,
     "kubeStateMetrics": {
      "metricLabelsAllowlist": "",
      "metricAnnotationsAllowList": ""
     }
    }
   },
   "resourceUID": "61534e2abb9a47000142c670"
  },
  "identity": {
   "type": "SystemAssigned",
   "principalId":  [redacted],
   "tenantId":  [redacted]
  },
  "sku": {
   "name": "Base",
   "tier": "Free"
  }
 }
cli.azure.cli.command_modules.acs.base_decorator: The intermediate 'subscription_id' does not exist. Return default value 'None'.
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/command_modules/acs/custom.py", line 767, in aks_update
    mc = aks_update_decorator.update_mc_profile_default()
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/command_modules/acs/managed_cluster_decorator.py", line 7299, in update_mc_profile_default
    mc = self.update_outbound_type_in_network_profile(mc)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/command_modules/acs/managed_cluster_decorator.py", line 6572, in update_outbound_type_in_network_profile
    raise InvalidArgumentValueError("Invalid outbound type, supported values are loadBalancer,"
azure.cli.core.azclierror.InvalidArgumentValueError: Invalid outbound type, supported values are loadBalancer, managedNATGateway, userAssignedNATGateway and userDefinedRouting.

cli.azure.cli.core.azclierror: Invalid outbound type, supported values are loadBalancer, managedNATGateway, userAssignedNATGateway and userDefinedRouting.
az_command_data_logger: Invalid outbound type, supported values are loadBalancer, managedNATGateway, userAssignedNATGateway and userDefinedRouting.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fc60cf06a20>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 1.273 seconds (init: 0.310, invoke: 0.963)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3657 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/local/pipx/venvs/azure-cli/bin/python /usr/local/pipx/venvs/azure-cli/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /root/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

As I have done on other clusters, I would expect this to update the outbound type and nat gateway outbound ip count accordingly.

Environment Summary

azure-cli 2.55.0

core 2.55.0 telemetry 1.1.0

Dependencies: msal 1.24.0b2 azure-mgmt-resource 23.1.0b2

Python location '/usr/local/pipx/venvs/azure-cli/bin/python' Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.7 (main, Dec 5 2023, 18:55:16) [GCC 12.2.0]

Additional context

No response

yonzhan commented 11 months ago

Thank you for opening this issue, we will look into it.

microsoft-github-policy-service[bot] commented 11 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dyu1208, @FumingZhang, @andyliuliming.

dsiperek-vendavo commented 8 months ago

We had the same issue. Turns out you most likely will need to run userAssignedNATGateway as you are doing BYO VNet.

FumingZhang commented 7 months ago

@xec-abailey it's not supported to migrate the outbound type from loadBalancer to managedNATGateway if the cluster is using BYO vnet.

https://learn.microsoft.com/en-us/azure/aks/egress-outboundtype#updating-outboundtype-after-cluster-creation