Open LiliDeng opened 2 months ago
Thank you for opening this issue, we will look into it.
@jiasli After changing azure-identity-1.9.0 into azure-identity-1.16.0, I saw below error. But I used the MSI with federated tokens have higher time limit. https://aka.ms/azdo-rm-workload-identity-manual, it is quite urgent for us, since we need get rid of the client secret of service principle.
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 197, in send
response = self.next.send(request)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\core\pipeline\policies\_retry.py", line 531, in send
response = self.next.send(request)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\core\pipeline\policies\_authentication.py", line 124, in send
self.on_request(request)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\core\pipeline\policies\_authentication.py", line 99, in on_request
self._token = self._credential.get_token(*self._scopes)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\identity\_credentials\default.py", line 219, in get_token
token = self._successful_credential.get_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\identity\_internal\decorators.py", line 33, in wrapper
token = fn(*args, **kwargs)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 112, in get_token
output = _run_command(command, self._process_timeout)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 215, in _run_command
raise ClientAuthenticationError(message=message) from ex
azure.core.exceptions.ClientAuthenticationError: ERROR: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-05-09T02:10:40.3190445Z, assertion valid from 2024-05-09T01:55:09.0000000Z, expiry time of assertion 2024-05-09T02:05:09.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 7ea7efe9-2c80-47d7-b82b-b51b5902be00 Correlation ID: 25cf5e33-3808-473b-9266-3748d846bedd Timestamp: 2024-05-09 02:10:40Z
Interactive authentication is needed. Please run:
az login
after upgrade az cli into 2.60.0 by running az upgrade --yes
in task: AzureCLI@2
, see a new issue
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: Authority.__init__() got an unexpected keyword argument 'oidc_authority_url'
Traceback (most recent call last):
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 664, in execute
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 731, in _run_jobs_serially
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 701, in _run_job
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 334, in __call__
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/profile/custom.py", line 97, in account_clear
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/_profile.py", line 296, in logout_all
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/auth/identity.py", line 210, in logout_all_users
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/auth/identity.py", line 119, in _msal_app
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msal/application.py", line 1827, in __init__
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msal/application.py", line 516, in __init__
TypeError: Authority.__init__() got an unexpected keyword argument 'oidc_authority_url'
The call stack provided in https://github.com/Azure/azure-cli/issues/28915#issue-2285238649 is trimmed. Please share the full call stack.
But I used the MSI with federated tokens have higher time limit.
It doesn't matter if the access token of a managed identity lasts longer. The OIDC token itself expires after 10 minuts:
Current time: 2024-05-09T02:10:40.3190445Z, assertion valid from 2024-05-09T01:55:09.0000000Z, expiry time of assertion 2024-05-09T02:05:09.0000000Z.
This issue is tracked by https://github.com/Azure/azure-cli/issues/28737.
TypeError: Authority.__init__() got an unexpected keyword argument 'oidc_authority_url'
I guess you are hitting https://github.com/Azure/azure-cli/issues/20990.
We are facing this issue on AzureCLI@2 task, (which currently uses Azure CLI 2.60.0 and masl 1.28.0) Also, wanted to know if there is any quick workaround we can do to fix it temporarily @jiasli ?
Traceback On pipeline
2024-05-07T14:48:56.0522913Z Traceback (most recent call last):
2024-05-07T14:48:56.0523329Z File "/home/vsts/work/1/s/lisa/lisa/testsuite.py", line 798, in __run_case
2024-05-07T14:48:56.0523678Z _call_with_retry_and_timeout(
2024-05-07T14:48:56.0524029Z File "/home/vsts/work/1/s/lisa/lisa/testsuite.py", line 64, in _call_with_retry_and_timeout
2024-05-07T14:48:56.0524366Z retry_call(
2024-05-07T14:48:56.0525015Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/retry/api.py", line 101, in retry_call
2024-05-07T14:48:56.0525468Z return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger)
2024-05-07T14:48:56.0526048Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/retry/api.py", line 33, in __retry_internal
2024-05-07T14:48:56.0526377Z return f()
2024-05-07T14:48:56.0526877Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/func_timeout/dafunc.py", line 108, in func_timeout
2024-05-07T14:48:56.0527293Z raise_exception(exception)
2024-05-07T14:48:56.0527806Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/func_timeout/py3_raise.py", line 7, in raise_exception
2024-05-07T14:48:56.0528154Z raise exception[0] from None
2024-05-07T14:48:56.0528473Z File "/home/vsts/work/1/s/lisa/lisa/testsuite.py", line 519, in wrapper
2024-05-07T14:48:56.0529172Z func(*args, **parameters)
2024-05-07T14:48:56.0529510Z File "/home/vsts/work/1/s/lisa/microsoft/testsuites/core/serial_console.py", line 41, in verify_serial_console
2024-05-07T14:48:56.0529899Z _ = serial_console.read()
2024-05-07T14:48:56.0530416Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
2024-05-07T14:48:56.0530842Z return caller(func, *(extras + args), **kw)
2024-05-07T14:48:56.0531663Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/retry/api.py", line 73, in retry_decorator
2024-05-07T14:48:56.0532570Z return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
2024-05-07T14:48:56.0533202Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/retry/api.py", line 33, in __retry_internal
2024-05-07T14:48:56.0533586Z return f()
2024-05-07T14:48:56.0533917Z File "/home/vsts/work/1/s/lisa/lisa/sut_orchestrator/azure/features.py", line 338, in read
2024-05-07T14:48:56.0534246Z output = self._read()
2024-05-07T14:48:56.0534678Z File "/home/vsts/work/1/s/lisa/lisa/sut_orchestrator/azure/features.py", line 376, in _read
2024-05-07T14:48:56.0535093Z self._initialize_serial_console(port_id=self.DEFAULT_SERIAL_PORT_ID)
2024-05-07T14:48:56.0535496Z File "/home/vsts/work/1/s/lisa/lisa/sut_orchestrator/azure/features.py", line 454, in _initialize_serial_console
2024-05-07T14:48:56.0536606Z serial_ports = self._serial_port_operations.list(
2024-05-07T14:48:56.0537432Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/mgmt/serialconsole/operations/_serial_ports_operations.py", line 101, in list
2024-05-07T14:48:56.0537947Z pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
2024-05-07T14:48:56.0538488Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 230, in run
2024-05-07T14:48:56.0538894Z return first_node.send(pipeline_request)
2024-05-07T14:48:56.0539413Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 86, in send
2024-05-07T14:48:56.0540157Z response = self.next.send(request)
2024-05-07T14:48:56.0540667Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 86, in send
2024-05-07T14:48:56.0541039Z response = self.next.send(request)
2024-05-07T14:48:56.0541541Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 86, in send
2024-05-07T14:48:56.0541915Z response = self.next.send(request)
2024-05-07T14:48:56.0542200Z [Previous line repeated 2 more times]
2024-05-07T14:48:56.0544190Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
2024-05-07T14:48:56.0544584Z response = self.next.send(request)
2024-05-07T14:48:56.0545153Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
2024-05-07T14:48:56.0545574Z response = self.next.send(request)
2024-05-07T14:48:56.0546081Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
2024-05-07T14:48:56.0546470Z response = self.next.send(request)
2024-05-07T14:48:56.0547009Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
2024-05-07T14:48:56.0547376Z self.on_request(request)
2024-05-07T14:48:56.0547926Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
2024-05-07T14:48:56.0548339Z self._token = self._credential.get_token(*self._scopes)
2024-05-07T14:48:56.0549085Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/identity/_credentials/default.py", line 169, in get_token
2024-05-07T14:48:56.0549760Z token = self._successful_credential.get_token(*scopes, **kwargs)
2024-05-07T14:48:56.0565106Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/identity/_internal/decorators.py", line 32, in wrapper
2024-05-07T14:48:56.0567902Z token = fn(*args, **kwargs)
2024-05-07T14:48:56.0571239Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/identity/_credentials/azure_cli.py", line 73, in get_token
2024-05-07T14:48:56.0571753Z output = _run_command(command)
2024-05-07T14:48:56.0572390Z File "/home/vsts/work/1/s/lisa/.venv/lib/python3.10/site-packages/azure/identity/_credentials/azure_cli.py", line 147, in _run_command
2024-05-07T14:48:56.0572891Z raise CredentialUnavailableError(message=NOT_LOGGED_IN)
2024-05-07T14:48:56.0573471Z azure.identity._exceptions.CredentialUnavailableError: Please run 'az login' to set up an account
@jiasli we still encounter this issue with azure cli 2.60.0+ version
2024-05-22 01:35:14.827[140164337821504][ERROR] lisa. KustoAuthenticationError('AzCliTokenProvider', 'KustoClientError("Failed to obtain Az Cli token for '[https://xxxx.westus2.kusto.windows.net'.\nPlease](https://xxxx.westus2.kusto.windows.net'./nPlease) be sure AzCli version 2.3.0 and above is intalled.\nERROR: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-05-22T01:35:10.6782380Z, assertion valid from 2024-05-22T00:23:05.0000000Z, expiry time of assertion 2024-05-22T00:33:05.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 1dcd51d9-6eaf-4db1-9fcb-c8f7faf10100 Correlation ID: 9d77aff6-4439-42d8-b763-6706640482e7 Timestamp: 2024-05-22 01:35:10Z\nInteractive authentication is needed. Please run:\naz login\n")', '{'authority:': 'AzCliTokenProvider', 'kusto_uri': 'https://xxxx.westus2.kusto.windows.net'}'/)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/mnt/vss/_work/1/s/lisa/lisa/util/parallel.py", line 58, in __call__
output = self._task()
File "/mnt/vss/_work/1/s/extensions/lsg_kusto.py", line 244, in _received_message
self._process_test_result_message(message)
File "/mnt/vss/_work/1/s/extensions/lsg_kusto.py", line 342, in _process_test_result_message
self._client.ingest_from_stream(stream, self._test_result_ingestion_props)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/ingest/ingest_client.py", line 74, in ingest_from_stream
containers = self._get_containers()
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/ingest/ingest_client.py", line 107, in _get_containers
containers = self._resource_manager.get_containers()
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/ingest/_resource_manager.py", line 140, in get_containers
self._refresh_ingest_client_resources()
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/ingest/_resource_manager.py", line 96, in _refresh_ingest_client_resources
self._ingest_client_resources = self._get_ingest_client_resources_from_service()
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/ingest/_resource_manager.py", line 103, in _get_ingest_client_resources_from_service
result = self._retryer(self._kusto_client.execute, "NetDefaultDB", ".get ingestion resources")
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 467, in __call__
do = self.iter(retry_state=retry_state)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 368, in iter
result = action(retry_state)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 390, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 470, in __call__
result = fn(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/data/client.py", line 138, in execute
return self.execute_mgmt(database, query, properties)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/data/client.py", line 163, in execute_mgmt
return self._execute(self._mgmt_endpoint, database, query, None, self._mgmt_default_timeout, properties)
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/data/client.py", line 233, in _execute
request_headers["Authorization"] = self._aad_helper.acquire_authorization_header()
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/azure/kusto/data/security.py", line 76, in acquire_authorization_header
raise KustoAuthenticationError(self.token_provider.name(), error, **kwargs)
azure.kusto.data.exceptions.KustoAuthenticationError: KustoAuthenticationError('AzCliTokenProvider', 'KustoClientError("Failed to obtain Az Cli token for '[https://xxxxxx.westus2.kusto.windows.net'.\nPlease](https://xxxxxx.westus2.kusto.windows.net'./nPlease) be sure AzCli version 2.3.0 and above is intalled.\nERROR: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-05-22T01:35:10.6782380Z, assertion valid from 2024-05-22T00:23:05.0000000Z, expiry time of assertion 2024-05-22T00:33:05.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 1dcd51d9-6eaf-4db1-9fcb-c8f7faf10100 Correlation ID: 9d77aff6-4439-42d8-b763-6706640482e7 Timestamp: 2024-05-22 01:35:10Z\nInteractive authentication is needed. Please run:\naz login\n")', '{'authority:': 'AzCliTokenProvider', 'kusto_uri': 'https://xxxxxx.westus2.kusto.windows.net'}'/)
@realsdx, can you get the raw error from Azure CLI?
@LiliDeng, does your task try to get an access token for https://xxxx.westus2.kusto.windows.net
after 5/10 minutes? If so, please see https://github.com/Azure/azure-cli/issues/28708#issuecomment-2047256166.
@jiasli we don't get the token in the task, we invoke python code to initiate the QueuedIngestClient with with_az_cli_authentication.
connection_string = KustoConnectionStringBuilder.with_az_cli_authentication( runbook.cluster ) self._client = QueuedIngestClient(connection_string)
@LiliDeng, it is possible QueuedIngestClient
is internally calling az account get-access-token
to get an access token for https://xxxx.westus2.kusto.windows.net
.
I have used below code for temp workaround.
credentials = DefaultAzureCredential()
connection_string = (
KustoConnectionStringBuilder.with_aad_application_token_authentication(
runbook.cluster,
credentials.get_token(
"https://kusto.kusto.windows.net/.default"
).token,
)
)
we used managed identity, the token expired time is 24 hours.
Describe the bug
We use below task in pipeline file.
And in the inlineScript we launch command to run python code, we always can see below error, I am sure the service connection has enough permission to operate the resources in the azure subscription.
Related command
'az account get-access-token --output json --resource'
Errors
see above
Issue script & Debug output
see above
Expected behavior
see above
Environment Summary
WARNING: You have 2 update(s) available. Consider updating your CLI installation with 'az upgrade' azure-cli 2.59.0 *
core 2.59.0 * telemetry 1.1.0
Extensions: azure-devops 1.0.0
Dependencies: msal 1.27.0 azure-mgmt-resource 23.1.0b2
Additional context
No response