bug: azure-cli 2.60+ - ssh fails using keys generated with az ssh config when run multiple times with service principal authentication

Describe the bug

I have reproduced this issue on both Ubuntu20.04 and RHEL8 images from the Azure Marketplace; as well as Microsoft hosted Azure DevOps Ubuntu Runners.

These versions work: 2.55.0, 2.56.0, 2.57.0, 2.58.0, 2.59.0

Broken: 2.60.0, 2.61.0, 2.62.0

Steps:

Deploy two target Linux VMs in Azure:
- Image: RHELRaw8LVMGen2
- Identity: SystemAssignedIdentity
- No public IP address (private only)
- Install the AADSSHloginForLinux VM extension
Create a new EntraID service principal. Grant the service principal the "virtual machine administrator login" role assignment on each virtual machine.
Deploy an ubuntu control host to test using Azure CLI to connect:
- Image: Ubuntu2004 (note, this can be reproduced also on the RHELRaw8LVMGen2 market place image)
- No public IP address

Install azure cli, and signin on the control host:

apt install azure-cli
az login --service-principal --allow-no-subscriptions --tenant <YOUR TENANT> --username <YOUR SP APPID>

Run this shell script:

az extension add --name ssh
az ssh config --file ~/.ssh/config --ip <Private IP of first target host>
az ssh config --file ~/.ssh/config --ip <Private IP of second target host>
ssh -v -o "StrictHostKeyChecking no" <Private IP of first target host> "echo hello first host!"
ssh -v -o "StrictHostKeyChecking no" <Private IP of second target host> "echo hello second host!"

Observe whichever host is configured first using az ssh config works; whichever is configured second will fail with this error:

debug1: sign_and_send_pubkey: no separate private key for certificate "/home/<my user>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub"
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/<my user>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub' are too open.

The the second host will always fail.

If you switch the order of the az ssh config commands, the opposite host will fail.

If you revert to Azure-cli 2.59.0 or older, this issue is not present:

        AZCLI_VER=2.59.0
        sudo apt remove azure-cli
        sudo apt-get install azure-cli=${AZCLI_VER}-1~$(lsb_release -cs)

Related command

az ssh config

Errors

The command completes successfully, however the key does not work for which whichever host has keys created second.

debug1: sign_and_send_pubkey: no separate private key for certificate "/home/<my user>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub"
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/<my user>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub' are too open.

Issue script & Debug output

cli.knack.cli: Command arguments: ['ssh', 'config', '--file', '/home/<MY USER>/.ssh/config', '--ip', '<MY IP>', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fb4d914a310>, <function OutputProducer.on_global_arguments at 0x7fb4d8e61ee0>, <function CLIQuery.on_global_arguments at 0x7fb4d8bf44c0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'ssh': ['azext_ssh']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: ssh                       0.158         1         4  /home/<MY USER>/.azure/cliextensions/ssh
cli.azure.cli.core: Total (1)                 0.158         1         4  
cli.azure.cli.core: Loaded 1 groups, 4 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : ssh config
cli.azure.cli.core: Command table: ssh config
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fb4d5546ee0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/<MY USER>/.azure/commands/2024-07-12.22-19-54.ssh_config.64974.log'.
az_command_data_logger: command args: ssh config --file {} --ip {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7fb4d54edaf0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7fb4d5511af0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7fb4d54b3940>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fb4d8e61f70>, <function CLIQuery.handle_query_parameter at 0x7fb4d8bf4550>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fb4d54b38b0>]
az_command_data_logger: extension name: ssh
az_command_data_logger: extension version: 2.0.4
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ComputeManagementClient
cli.azure.cli.core.auth.persistence: build_persistence: location='/home/<MY USER>/.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/home/<MY USER>/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /home/<MY USER>/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/<MY TENANT>
msal.authority: openid_config("https://login.microsoftonline.com/<MY TENANT>/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/<MY TENANT>/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/<MY TENANT>/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azext_ssh.ssh_utils: Running ssh-keygen command ssh-keygen -f /home/<MY USER>/.ssh/az_ssh_config/<MY IP>/id_rsa -t rsa -q -N 
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/<MY TENANT>
msal.authority: openid_config("https://login.microsoftonline.com/<MY TENANT>/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/<MY TENANT>/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/<MY TENANT>/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/<MY TENANT>/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes= # REMOVED
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://pas.windows.net/CheckMyAccess/Linux/.default',), # REMOVED
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 5ac412e3-9ba7-4251-9adc-b72c376956fd
cli.azext_ssh.custom: Generating certificate /home/<MY USER>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub
cli.azext_ssh.ssh_utils: Running ssh-keygen command ssh-keygen -L -f /home/<MY USER>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub
cli.azext_ssh.ssh_utils: Running ssh-keygen command ssh-keygen -L -f /home/<MY USER>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub
Generated SSH certificate /home/<MY USER>/.ssh/az_ssh_config/<MY IP>/id_rsa.pub-aadcert.pub is valid until 2024-07-12 11:01:38 PM in local time.
cli.azext_ssh.ssh_utils: /home/<MY USER>/.ssh/az_ssh_config/<MY IP> contains sensitive information (id_rsa, id_rsa.pub, id_rsa.pub-aadcert.pub). Please delete it once you no longer need this config file.
cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7fb4d550cdc0>, <function _x509_from_base64_to_hex_transform at 0x7fb4d550ce50>]
cli.knack.cli: Event: CommandInvoker.OnFilterResult []
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fb4d5547160>]
az_command_data_logger: exit code: 0
cli.__main__: Command ran in 0.792 seconds (init: 0.151, invoke: 0.641)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3974 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/bin/python3.9 /usr/lib64/az/lib/python3.9/site-packages/azure/cli/telemetry/__init__.py /home/<MY USER>/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

I can set up ssh keys for multiple hosts using az ssh config on versions newer than azure-cli 2.60; as works in 2.59.0 and older.

Environment Summary

azure-cli                         2.62.0

core                              2.62.0
telemetry                          1.1.0

Extensions:
ssh                                2.0.4

Dependencies:
msal                              1.28.1
azure-mgmt-resource               23.1.1

Additional context

No response

Hi @rybal06 Find similar issue https://github.com/Azure/azure-cli/issues/29006.
Issue title	az ssh arc connection issue using Service Principal
Create time	2024-05-22
Comment number	4

Azure / azure-cli