Failed to add Load Balancer addresses to backend pools when syncMode of the Backend pool is set to 'Automatic'

Describe the bug

When creating the backend pool setting the property 'sync-mode' to 'Automatic' and then adding an address to the pool is failing due of next error:

"Error: updating Backend Address Pool Address: (Address Name "<backend_pool_address_name>" / Backend Address Pool Name "<backend_pool_name>" / Load Balancer Name "<load_balancer_name>" / Resource Group "<resource_group_name>"): performing LoadBalancerBackendAddressPoolsCreateOrUpdate: unexpected status 400 (400 Bad Request) with error: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.

Related command

az network lb address-pool create--lb-name lbname -g rgname -n bpname--sync-mode "Automatic" --vnet "/subscriptions/<az_sub_id>/resourceGroups/rgname/providers/Microsoft.Network/virtualNetworks/vnetname"

az network lb address-pool address add --ip-address "10.0.1.10" --lb-name lbname -g rgname --name bpaddress_name --pool-name bpname --virtual-network "/subscriptions/<az_sub_id>/resourceGroups/rgname/providers/Microsoft.Network/virtualNetworks/vnetname"

Errors

Error: updating Backend Address Pool Address: (Address Name "<backend_pool_address_name>" / Backend Address Pool Name "<backend_pool_name>" / Load Balancer Name "<load_balancer_name>" / Resource Group "<resource_group_name>"): performing LoadBalancerBackendAddressPoolsCreateOrUpdate: unexpected status 400 (400 Bad Request) with error: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.

Issue script & Debug output

$> az network lb address-pool address add --ip-address "10.0.1.10" --lb-name lbname -g rsgname --name Testing_BackendPool --pool-name Testing_BackendPool --virtual-network "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name" --debug

cli.knack.cli: Command arguments: ['network', 'lb', 'address-pool', 'address', 'add', '--ip-address', '10.0.1.10', '--lb-name', 'lbname', '-g', 'rsgname', '--name', 'Testing_BackendPool', '--pool-name', 'Testing_BackendPool', '--virtual-network', '/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x014E2A78>, <function OutputProducer.on_global_arguments at 0x0180AB68>, <function CLIQuery.on_global_arguments at 0x0182C938>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'network': ['azure.cli.command_modules.network', 'azure.cli.command_modules.privatedns']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: network                   0.892       124       488
cli.azure.cli.core: privatedns                0.051        14        60
cli.azure.cli.core: Total (2)                 0.943       138       548
cli.azure.cli.core: Loaded 137 groups, 548 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : network lb address-pool address add
cli.azure.cli.core: Command table: network lb address-pool address add
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x03CEC258>]    
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\path\.azure\commands\2024-08-14.12-54-21.network_lb_address-pool_address_add.9476.log'.
az_command_data_logger: command args: network lb address-pool address add --ip-address {} --lb-name {} -g {} --name {} --pool-name {} --virtual-network {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x03D1D3E8>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x03D306B8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x03D30708>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0180ABB8>, <function CLIQuery.handle_query_parameter at 0x0182C988>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x03D30668>]
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\\Users\\path\\.azure\\service_principal_entries.bin', encrypt=True
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\\Users\\path\\.azure\\msal_token_cache.bin', encrypt=True
cli.azure.cli.core.auth.binary_cache: load: C:\Users\path\.azure\msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/<tenant_id>
msal.authority: openid_config("https://login.microsoftonline.com/<tenant_id>/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/<tenant_id>/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/<tenant_id>/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 
'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/<tenant_id>/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), 
kwargs={}
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.telemetry: Generate or reuse correlation_id: cc265b56-7579-48fc-86b3-1db241262a37
urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443
urllib3.connectionpool: https://login.microsoftonline.com:443 "POST /<tenant_id>/oauth2/v2.0/token HTTP/1.1" 200 1499
msal.token_cache: event={
    "client_id": "<client_id>",
    "data": {
        "claims": "{\"access_token\": {\"xms_cc\": {\"values\": [\"CP1\"]}}}",
        "scope": [
            "https://management.core.windows.net//.default"
        ]
    },
    "environment": "login.microsoftonline.com",
    "grant_type": "client_credentials",
    "params": null,
    "response": {
        "access_token": "********",
        "expires_in": 3599,
        "ext_expires_in": 3599,
        "token_type": "Bearer"
    },
    "scope": [
        "https://management.core.windows.net//.default"
    ],
    "token_endpoint": "https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token"
}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': ''
cli.azure.cli.core.sdk.policies:     'CommandName': 'network lb address-pool address add'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--ip-address --lb-name -g --name --pool-name --virtual-network --debug'   
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.61.0 (MSI) azsdk-python-core/1.28.0 Python/3.11.8 (Windows-10-10.0.19045-SP0)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01 HTTP/1.1" 200 935
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '935'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'ETag': 'W/"2e7b4"'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': 'f0fe788f-b59e8d02'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '69301b4ba2b6e'
cli.azure.cli.core.sdk.policies:     'x-ms-arm-service-request-id': '16d6194c3b7926'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '249'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-global-reads': '3749'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'FRANCESOUTH:202a2b6e' 
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: E25E733 Ref C: 2024-08-14T10:54:23Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Wed, 14 Aug 2024 10:54:23 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
  "name": "Testing_BackendPool",
  "id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool",
  "etag": "W/\"2b3dcb17b4\"",
  "properties": {
    "provisioningState": "Succeeded",
    "virtualNetwork": {
      "id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name"
    },
    "loadBalancerBackendAddresses": [],
    "loadBalancingRules": [
      {
        "id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/loadBalancingRules/Rule_3389"
      }
    ],
    "syncMode": "Automatic"
  },
  "type": "Microsoft.Network/loadBalancers/backendAddressPools"
}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01'
cli.azure.cli.core.sdk.policies: Request method: 'PUT'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'Content-Length': '730'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '8e9'
cli.azure.cli.core.sdk.policies:     'CommandName': 'network lb address-pool address add'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--ip-address --lb-name -g --name --pool-name --virtual-network --debug'   
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.61.0 (MSI) azsdk-python-core/1.28.0 Python/3.11.8 (Windows-10-10.0.19045-SP0)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: {"id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool", "name": "Testing_BackendPool", "properties": {"loadBalancerBackendAddresses": [{"name": "Testing_BackendPool", "properties": {"ipAddress": "10.0.1.10", "virtualNetwork": {"id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name"}}}], "syncMode": "Automatic", "virtualNetwork": {"id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name"}}}
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01 HTTP/1.1" 400 247
cli.azure.cli.core.sdk.policies: Response status: 400
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '247'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '60c870c9514'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '78ef36a'
cli.azure.cli.core.sdk.policies:     'x-ms-arm-service-request-id': '7fad'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-writes': '199'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-global-writes': '2999'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'FRANCESOUTH:2021e9c6a' 
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: B03B6B8023 Ref C: 2024-08-14T10:54:24Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Wed, 14 Aug 2024 10:54:23 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
  "error": {
    "code": "ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic",
    "message": "Modification of backend addresses in pools is not supported when sync mode is set as Automatic.",
    "details": []
  }
}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 664, in execute
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 731, in _run_jobs_serially
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 712, in _run_job
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 1052, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 1039, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 108, in result    
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 130, in wait      
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 83, in _start     
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure\cli\command_modules\network\aaz_compact\latest\network\lb\address_pool\address\__cmds.py", line 118, in _execute_operations
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure\cli\command_modules\network\aaz_compact\latest\network\lb\address_pool\address\__cmds.py", line 278, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_operation.py", line 332, in on_error
azure.core.exceptions.HttpResponseError: (ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic) Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
Code: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic
Message: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.

cli.azure.cli.core.azclierror: (ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic) Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
Code: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic
Message: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
az_command_data_logger: (ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic) Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
Code: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic
Message: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x03CEC398>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 4.774 seconds (init: 0.712, invoke: 4.062)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4242 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry\__init__.pyc C:\Users\path\.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

Adding addresses to the backend pool that it´s previously created should work as it works when the backend pool sync_mode property is set to 'Manual'. It appears that addresses can´t be added when sync mode of the backend pool is set to 'Automatic' but there´s nothing about it in the limitations doc https://learn.microsoft.com/en-us/azure/load-balancer/cross-subscription-overview#limitations

Environment Summary

azure-cli 2.61.0 *

core 2.61.0 * telemetry 1.1.0

Dependencies: msal 1.28.0 azure-mgmt-resource 23.1.1

Python (Windows) 3.11.8 (tags/v3.11.8:db85d51, Feb 6 2024, 21:52:07) [MSC v.1937 32 bit (Intel)]

Additional context

Also the alternative way creating the backend pool without sync_mode and adding the address to the pool so then changing the backend pool sync_mode property to 'Automatic' does not work:

(SyncModePropertyCannotBetSetWithLoadBalancerBackendAddressesPresent) Sync mode property of load balancer backend address pool cannot be set with backend addresses present in the pool. Code: SyncModePropertyCannotBetSetWithLoadBalancerBackendAddressesPresent Message: Sync mode property of load balancer backend address pool cannot be set with backend addresses present in the pool.

Azure / azure-cli