fausttiger007
commented
1 week ago OK... works fine with Automatic Serverless Runtime options... with existing Compute Instance it is failing now (use to work).
I noticed that similar (error message, but different prompt flow module) issue https://github.com/microsoft/promptflow/issues/973 referenced compute instance /identity related issue (not in detail), I tried with the Auto runtime.
Still not sure why Compute Instances fail, but are looking into that.
Describe the bug In azure prompt flow... using standard Q&A flow with history (which has been running fine)... it now fails with error listed below... under these 2 scenarios 1) when executing lookup_question_from_indexed_docs node, using "Index Lookup" tool 2) when attempting to edit fields of mlindex_context of new flow (which defines which instances used of OpenAI, AI Search, etc.)
Note ONLY happening in SouthCentralUS region... West Europe works fine.
Have been using for months in both WestEurope and SouthCentralUS, problem only started recently in SouthCentralUS. Endpoint deployed off this flow in SouthCentralUS still works.
Connections to DefaultAzureOpenAI work... this is related to "lookup Index" tool and possibly Azure AI Search connection.
"Execution failure in 'lookup_question_from_indexed_docs': (ClientAuthenticationError) DefaultAzureCredential failed to retrieve a token from the included credentials.\nAttempted credentials:\n\tEnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.\nVisit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.\n\tManagedIdentityCredential: Unexpected content type \"text/plain; charset=utf-8\"\nContent: SSO failure: unable to retrieve a token.
DETAILED OUTPUT TRACE "status": "Failed" "error": { "additionalInfo": [ 0: { "info": { ... } "type": "ToolExecutionErrorDetails" } ] "code": "UserError" "debugInfo": { "innerException": { "innerException": null "message": "DefaultAzureCredential failed to retrieve a token from the included credentials.\nAttempted credentials:\n\tEnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.\nVisit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.\n\tManagedIdentityCredential: Unexpected content type \"text/plain; charset=utf-8\"\nContent: SSO failure: unable to retrieve a token. Error \nTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot." "stackTrace": "Traceback (most recent call last):\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py\", line 190, in _invoke_tool_with_timer\n return f(kwargs)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/tracer.py\", line 458, in wrapped\n output = func(*args, *kwargs)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow_vectordb/tool/common_index_lookup.py\", line 58, in search\n search_func = _get_search_func(mlindex_content, top_k, query_type)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow_vectordb/tool/common_index_lookup.py\", line 37, in _get_search_func\n return build_search_func(index, top_k, query_type)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow_vectordb/tool/common_index_lookup_extensions/utils.py\", line 24, in build_search_func\n store = index.as_langchain_vectorstore()\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azureml/rag/mlindex.py\", line 179, in as_langchain_vectorstore\n raise e\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azureml/rag/mlindex.py\", line 169, in as_langchain_vectorstore\n connection_credential = get_connection_credential(self.index_config, credential=credential)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azureml/rag/utils/connections.py\", line 75, in get_connection_credential\n connection = get_connection_by_id_v2(connection_id, credential=credential)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azureml/rag/utils/connections.py\", line 219, in get_connection_by_id_v2\n list_secrets_response = ml_client.connections._operation.list_secrets(\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/tracing/decorator.py\", line 78, in wrapper_use_tracer\n return func(args, kwargs)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/ai/ml/_restclient/v2023_08_01_preview/operations/_workspace_connections_operations.py\", line 998, in list_secrets\n pipeline_response: PipelineResponse = self._client._pipeline.run( # pylint: disable=protected-access\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/_base.py\", line 230, in run\n return first_node.send(pipeline_request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/_base.py\", line 86, in send\n response = self.next.send(request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/_base.py\", line 86, in send\n response = self.next.send(request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/_base.py\", line 86, in send\n response = self.next.send(request)\n [Previous line repeated 2 more times]\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/mgmt/core/policies/_base.py\", line 46, in send\n response = self.next.send(request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect.py\", line 197, in send\n response = self.next.send(request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry.py\", line 531, in send\n response = self.next.send(request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py\", line 124, in send\n self.on_request(request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py\", line 99, in on_request\n self._token = self._credential.get_token(self._scopes)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/azure/identity/_credentials/default.py\", line 225, in get_token\n token = super().get_token(scopes, claims=claims, tenant_id (truncated, 4.2K) "type": "ClientAuthenticationError" } "message": "Execution failure in 'lookup_question_from_indexed_docs': (ClientAuthenticationError) DefaultAzureCredential failed to retrieve a token from the included credentials.\nAttempted credentials:\n\tEnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.\nVisit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.\n\tManagedIdentityCredential: Unexpected content type \"text/plain; charset=utf-8\"\nContent: SSO failure: unable to retrieve a token. Error \nTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot." "stackTrace": "\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py\", line 88, in invoke_tool\n result = self._invoke_tool_with_timer(node, f, kwargs)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py\", line 201, in _invoke_tool_with_timer\n raise ToolExecutionError(node_name=node_name, module=module) from e\n" "type": "ToolExecutionError" } "innerError": { "code": "ToolExecutionError" "innerError": null } "message": "Execution failure in 'lookup_question_from_indexed_docs': (ClientAuthenticationError) DefaultAzureCredential failed to retrieve a token from the included credentials.\nAttempted credentials:\n\tEnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.\nVisit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.\n\tManagedIdentityCredential: Unexpected content type \"text/plain; charset=utf-8\"\nContent: SSO failure: unable to retrieve a token. Error \nTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot." "messageFormat": "Execution failure in '{node_name}'." "messageParameters": { "node_name": "lookup_question_from_indexed_docs" } "referenceCode": "Tool/promptflow_vectordb.tool.common_index_lookup" } "output": null
How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug: 1.when executing lookup_question_from_indexed_docs node, using "Index Lookup" tool
Expected behavior The error messages above... see screenshots below
Screenshots
.
.
Running Information(please complete the following information): This is using the AzureML Portal instance/UI in ML Workspace, and not a separate local Python installation. This is Prompt Flow in Azure ML Workspace, and NOT prompt flow in AI Studio.
pf -v: [e.g. 0.0.102309906] don't know... whatever is current for ML Workspace currently for SouthCentralUSpython --version: [e.g. python==3.10.12] error messages indicate Python 3.9 /azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.pyAdditional context Some similarity to SSO failure: unable to retrieve a token · Issue #973 · microsoft/promptflow (github.com) although different part of prompt flow, and only in SouthCentralUS Only started happening recently in SouthCentralUS region... works fine in West Europe.