search

Azure-Samples / ai-rag-chat-evaluator

Tools for evaluation of RAG Chat Apps using Azure AI Evaluate SDK and OpenAI
MIT License
162 stars 58 forks source link

Credential failure on evaluate script #87

Open diberry opened 1 month ago

diberry commented 1 month ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

  1. azd up
  2. Look at .env
AZURE_ENV_NAME="dfberry-py-evaluations"
AZURE_LOCATION="swedencentral"
AZURE_OPENAI_EVAL_DEPLOYMENT="eval"
AZURE_OPENAI_RESOURCE_GROUP="dfberry-py-evaluationstrvkoypjkqie6-rg"
AZURE_OPENAI_SERVICE="dfberry-py-evaluationstrvkoypjkqie6-openai"
AZURE_RESOURCE_GROUP="dfberry-py-evaluationstrvkoypjkqie6-rg"
AZURE_SUBSCRIPTION_ID="2123cb08-23fa-4a80-afec-4ec9e015433f"
AZURE_TENANT_ID="72f988bf-86f1-41af-91ab-2d7cd011db47"
OPENAI_GPT_MODEL="gpt-4"
OPENAI_HOST="azure"
OPENAI_KEY=""
OPENAI_ORGANIZATION=""
  1. Run script: python3 -m scripts generate --output=my_input/qa.jsonl --numquestions=14 --persource=2
  2. Run Script: python3 -m scripts evaluate --config=my_config.json --numquestions=14 with the following my_config.json
{
    "testdata_path": "my_input/qa.jsonl",
    "results_dir": "my_results/experiment_refined",
    "target_url": "https://app-backend-uchshrwcihlhg.azurewebsites.net/chat",
    "target_parameters": {
        "overrides": {
            "semantic_ranker": false,
            "prompt_template": "<READFILE>my_input/prompt_refined.txt"
        }
    }
}
  1. Get the following error: 
diberry ➜ /workspaces/ai-rag-chat-evaluator (main) $ python3 -m scripts evaluate --config=my_config.json --numquestions=14
[22:43:56] INFO     Running evaluation from config                                               evaluate.py:207
                    /workspaces/ai-rag-chat-evaluator/my_config.json                                            
           INFO     Replaced prompt_template in config with contents of                          evaluate.py:201
                    my_input/prompt_refined.txt                                                                 
           INFO     Using Azure OpenAI Service with Azure Developer CLI Credential           service_setup.py:27
           INFO     Running evaluation using data from                                            evaluate.py:77
                    /workspaces/ai-rag-chat-evaluator/my_input/qa.jsonl                                         
           INFO     Limiting evaluation to 14 questions                                           evaluate.py:80
           INFO     Sending a test question to the target to ensure it is running...              evaluate.py:83
[22:44:43] INFO     Successfully received response from target for question: "What information is evaluate.py:92
                    in your knowledge base?"                                                                    
                    "answer": "Our knowledge base includes information on custome..."                           
                    "context": "role_library.pdf#page=31:  Responsibilities: · Mai..."                          
           INFO     Sending a test chat completion to the GPT deployment to ensure it is         evaluate.py:102
                    running...                                                                                  
[22:44:46] INFO     Successfully received response from GPT: "Hello! How can I assist you        evaluate.py:109
                    today?"                                                                                     
           INFO     Starting evaluation...                                                       evaluate.py:114
[22:45:00] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:00.062907884Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: ba0a99de-13ba-4ae6-975c-b465e1195500 Correlation ID:                              
                    6906592c-47f3-4e5b-9d23-8e221afc8ea6 Timestamp: 2024-05-13 22:45:00Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:01] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:01.462735824Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: f46f2ce6-c2df-41f5-8a8b-e288cbcd5600 Correlation ID:                              
                    a866b280-de86-489b-8b32-6decb8b920a1 Timestamp: 2024-05-13 22:45:01Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:02] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.186514156Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 0d826a6e-4544-4706-8e05-8cd318e55c00 Correlation ID:                              
                    c68a6fef-59ca-4916-b452-012072a747e2 Timestamp: 2024-05-13 22:45:02Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
           WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.420259563Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 90ec62ee-fc30-447e-a097-b8883cd25800 Correlation ID:                              
                    6a064df5-461c-4dce-9354-e552ce9fa798 Timestamp: 2024-05-13 22:45:02Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
           WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.593558324Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 9175eb02-a11c-464e-86cb-5642c8716600 Correlation ID:                              
                    64131b49-df44-4386-8781-83bd7688020b Timestamp: 2024-05-13 22:45:02Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:05] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:05.920895603Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 9ba8e8ea-8145-4f06-b234-8ae6a7b05e00 Correlation ID:                              
                    3583cdf4-9218-47a4-aa45-af7529d9c1ed Timestamp: 2024-05-13 22:45:05Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:06] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:06.9722591Z","data":{"               
                    message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)                 
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: f3bc8a18-e5f4-47c9-b4de-4c4ad1cd5400 Correlation ID:                              
                    4ea32ae6-f7b4-464c-a01a-ce3471caa562 Timestamp: 2024-05-13 22:45:06Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:09] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:09.183899943Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 72a4f6c6-e0c3-4fa7-8e23-c7cb14444f00 Correlation ID:                              
                    0d471dd1-5f8d-4e20-8193-5181930da6b5 Timestamp: 2024-05-13 22:45:09Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:10] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:10.152382316Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: da7b4698-7ccd-4c23-a971-1fa78dbc7400 Correlation ID:                              
                    332fe3da-4f22-4244-9a08-895c734ec079 Timestamp: 2024-05-13 22:45:10Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:14] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:14.587324371Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 812f3b95-f0d4-42f6-90f3-0a6a25a45b00 Correlation ID:                              
                    82126b7c-93c2-4c8e-a82d-5d98644160e4 Timestamp: 2024-05-13 22:45:14Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
           WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:14.629979902Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 2de505a7-0a08-4b9b-a33f-01ebc94e6f00 Correlation ID:                              
                    032e19b9-352d-421e-adc5-b59f83c7ed26 Timestamp: 2024-05-13 22:45:14Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
           WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:14.68679813Z","data":{               
                    "message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)                
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 2b6d94bd-5dfa-4de5-adbf-26eef5935000 Correlation ID:                              
                    fbb61a5c-3945-4c0b-b0c7-29d2c4209b4c Timestamp: 2024-05-13 22:45:14Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:16] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:16.094273693Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 9175eb02-a11c-464e-86cb-564244746600 Correlation ID:                              
                    d516d97b-614c-4c79-b401-16ed0c2d1701 Timestamp: 2024-05-13 22:45:16Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
[22:45:20] WARNING  DefaultAzureCredential failed to retrieve a token from the included           chained.py:123
                    credentials.                                                                                
                    Attempted credentials:                                                                      
                            EnvironmentCredential: EnvironmentCredential authentication                         
                    unavailable. Environment variables are not fully configured.                                
                    Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot               
                    to troubleshoot this issue.                                                                 
                            ManagedIdentityCredential: ManagedIdentityCredential authentication                 
                    unavailable. The requested identity has not been assigned to this resource.                 
                    Error: Unexpected response "{'error': 'invalid_request', 'error_description':               
                    'Identity not found'}"                                                                      
                            SharedTokenCacheCredential: SharedTokenCacheCredential authentication               
                    unavailable. No accounts were found in the cache.                                           
                            AzureCliCredential: Azure CLI not found on path                                     
                            AzurePowerShellCredential: PowerShell is not installed                              
                            AzureDeveloperCliCredential:                                                        
                    {"type":"consoleMessage","timestamp":"2024-05-13T22:45:20.531998191Z","data":               
                    {"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)               
                    AADSTS70011: The provided request must include a 'scope' input parameter. The               
                    provided value for the input parameter 'scope' is not valid. The scope                      
                    https://cognitiveservices.azure.com/ openid offline_access profile is not                   
                    valid. The scope format is invalid. Scope must be in a valid URI form                       
                    \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.                   
                    Trace ID: 451a61af-cf36-4f88-9194-1536ab515800 Correlation ID:                              
                    4c65d9ed-12db-4f71-9033-3b743c28135c Timestamp: 2024-05-13 22:45:20Z\n\n"}}                 

                    To mitigate this issue, please refer to the troubleshooting guidelines here                 
                    at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.                
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspaces/ai-rag-chat-evaluator/scripts/__main__.py", line 6, in <module>
    app()
  File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 326, in __call__
    raise e
  File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 309, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/typer/core.py", line 723, in main
    return _main(
           ^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/typer/core.py", line 193, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 692, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/ai-rag-chat-evaluator/scripts/cli.py", line 47, in evaluate
    run_evaluate_from_config(Path.cwd(), config, numquestions, targeturl)
  File "/workspaces/ai-rag-chat-evaluator/scripts/evaluate.py", line 214, in run_evaluate_from_config
    evaluation_run_complete = run_evaluation(
                              ^^^^^^^^^^^^^^^
  File "/workspaces/ai-rag-chat-evaluator/scripts/evaluate.py", line 153, in run_evaluation
    questions_with_ratings.append(future.result())
                                  ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/ai-rag-chat-evaluator/scripts/evaluate.py", line 135, in evaluate_row
    result = metric.evaluator_fn(openai_config=openai_config)(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/evals/evaluators/_groundedness/_groundedness.py", line 56, in __call__
    llm_output = self._flow(answer=answer, context=context)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/_sdk/entities/_flows/prompty.py", line 57, in __call__
    return self._core_prompty(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/tracing/_trace.py", line 469, in wrapped
    output = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/core/_flow.py", line 410, in __call__
    response = send_request_to_llm(api_client, self._model.api, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/core/_prompty_utils.py", line 164, in send_request_to_llm
    result = client.chat.completions.create(**parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/_utils/_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 590, in create
    return self._post(
           ^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/_base_client.py", line 939, in _request
    self._prepare_options(options)
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/lib/azure.py", line 288, in _prepare_options
    azure_ad_token = self._get_azure_ad_token()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/openai/lib/azure.py", line 274, in _get_azure_ad_token
    token = provider()
            ^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/core/_connection.py", line 247, in get_token
    return self._token_provider.get_token()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/_core/token_provider.py", line 52, in get_token
    return self.credential.get_token(audience).token
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/identity/_credentials/default.py", line 225, in get_token
    token = super().get_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/identity/_credentials/chained.py", line 124, in get_token
    raise ClientAuthenticationError(message=message)
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
        ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource. Error: Unexpected response "{'error': 'invalid_request', 'error_description': 'Identity not found'}"
        SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
        AzureCliCredential: Azure CLI not found on path
        AzurePowerShellCredential: PowerShell is not installed
        AzureDeveloperCliCredential: {"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.593558324Z","data":{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope) AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope https://cognitiveservices.azure.com/ openid offline_access profile is not valid. The scope format is invalid. Scope must be in a valid URI form \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e. Trace ID: 9175eb02-a11c-464e-86cb-5642c8716600 Correlation ID: 64131b49-df44-4386-8781-83bd7688020b Timestamp: 2024-05-13 22:45:02Z\n\n"}}

To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
  1. To fix, tried to add azd env for az cli: azd config set auth.useAzCliAuth "true" then azd auth login again and az cli isn't in repo.

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

Thanks! We'll be in touch soon.

pamelafox commented 1 month ago

Ah I saw that on CI. I'll try to replicate locally. It shouldnt need az CLI.

pamelafox commented 1 month ago

Replicated, darn. I was testing outside of the dev container, where I must have been logged in with the Azure CLI. The new Promptflow SDK calls DefaultAzureCredential behind the scenes, and is doing it in some way that's not working with azd login. I'm looking into it.

pamelafox commented 1 month ago

Sent PR to PromptFlow SDK: https://github.com/microsoft/promptflow/pull/3244

diberry commented 1 month ago

@pamelafox do I need to set azd config set auth.useAzCliAuth "true" before azd up?

pamelafox commented 1 month ago

No, you would need to use the actual Azure CLI to login, so that DefaultAzureCredential would use the Azure CLI credential. Promptflow's usage of DefaultAzureCredential isn't compatible with the Azure Developer CLI credential. The azd login is still fine for the azd up process.

My PR was merged, so I am awaiting the release of the new SDK. But I can add the Azure CLI to this container for now and a step to login, since that might take a few days.