Connection is not found during evaluate

[echen101]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

1. generate script works fine against index
2. using local running app with target url: http://127.0.0.1:50505/chat
3. python -m scripts evaluate --config=example_config.json --numquestions=2

Any log messages given by the failure

Traceback (most recent call last): File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/azure/ai/generative/evaluate/_evaluate.py", line 359, in _evaluate inbuilt_metrics_results = inbuilt_metrics_handler.calculate_metrics() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/azure/ai/generative/evaluate/_metric_handler.py", line 108, in calculate_metrics pf_run = run_pf_flow_with_dict_list(flow_path, dict_list, flow_params={"connections": {node: connection_override for node in nodes_list}}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/azure/ai/generative/evaluate/_utils.py", line 65, in run_pf_flow_with_dict_list return pf_client.run( ^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_pf_client.py", line 187, in run return self.runs.create_or_update(run=run, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_telemetry/activity.py", line 265, in wrapper return f(self, *args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/operations/_run_operations.py", line 103, in create_or_update created_run = RunSubmitter(client=self._client).submit(run=run, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_submitter/run_submitter.py", line 40, in submit self._run_bulk(run=run, stream=stream, *kwargs) File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_submitter/run_submitter.py", line 97, in _run_bulk self._submit_bulk_run(flow=flow, run=run, local_storage=local_storage) File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_submitter/run_submitter.py", line 118, in _submit_bulk_run connections = SubmitterHelper.resolve_connections(flow=flow) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_submitter/utils.py", line 276, in resolve_connections return SubmitterHelper.resolve_connection_names( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_submitter/utils.py", line 348, in resolve_connection_names raise e File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_submitter/utils.py", line 344, in resolve_connection_names conn = client.connections.get(name=n, with_secrets=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_telemetry/activity.py", line 265, in wrapper return f(self, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/operations/_connection_operations.py", line 52, in get return self._get(name, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/operations/_connection_operations.py", line 57, in _get orm_connection = ORMConnection.get(name, raise_error) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_orm/retry.py", line 43, in f_retry return f(args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/Users/hzwnfj/workspaces/ai-rag-chat-evaluator/.venv/lib/python3.12/site-packages/promptflow/_sdk/_orm/connection.py", line 52, in get raise ConnectionNotFoundError(f"Connection {name!r} is not found.") promptflow._sdk._errors.ConnectionNotFoundError: Connection 'Default_AzureOpenAI' is not found.

Expected/desired behavior

OS and Version?

macOS Sonoma 14.3.1

Versions

Mention any other details that might be useful

I've tried evaluating all questions (200) and a subset (2), both failed with the same connection error. The test question went fine as I saw the log before the error

14:59:06 (INFO) scripts: Sending a test chat completion to the GPT deployment to ensure it is running...
14:59:08 (INFO) httpx: HTTP Request: POST https://dig-openai.openai.azure.com/openai/deployments/bd-gpt4-32k/chat/completions?api-version=2023-07-01-preview "HTTP/1.1 200 OK"
14:59:08 (INFO) scripts: Successfully received response from GPT: Hello! How can I assist you today?
14:59:08 (INFO) scripts: Starting evaluation...

---

Thanks! We'll be in touch soon.

[mhauschild]

Having the same issue here. Which python version are you using? Downgraded my python venv from 3.12 to 3.10. Thought that this might be an issue with promptflow.

But now I don't get any useful error details anymore:

...\ai-rag-chat-evaluator> python -m scripts evaluate --config=example_config.json --numquestions=2 12:53:16 (INFO) scripts: Running evaluation from config ...\ai-rag-chat-evaluator\example_config.json 12:53:16 (INFO) scripts: Replaced results_dir in config with timestamp 12:53:16 (INFO) scripts: Replaced prompt_template in config with contents of example_input/prompt_feri.txt 12:53:16 (INFO) scripts: Using Azure OpenAI Service with API Key from AZURE_OPENAI_KEY 12:53:16 (INFO) scripts: {'api_type': 'azure', 'api_base': 'https://xxx.openai.azure.com', 'api_key': 'xxx', 'api_version': '2023-07-01-preview', 'deployment_id': 'chat', 'model': 'gpt-35-turbo'} 12:53:16 (INFO) scripts: Running evaluation using data from ...\ai-rag-chat-evaluator\example_input\feri_qa.jsonl 12:53:16 (INFO) scripts: Limiting evaluation to 2 questions 12:53:16 (INFO) scripts: Sending a test question to the target to ensure it is running... 12:53:16 (ERROR) scripts: Failed to send a test question to the target due to error: Expecting value: line 1 column 1 (char 0) 12:53:16 (ERROR) scripts: Evaluation was terminated early due to an error ⬆

[pamelafox]

@mhauschild That looks like a different issue, as its failing to get a JSON response from your chat endpoint. The target URL in example_config.json must correspond to the endpoint that receives POST requests from the chat app, like https://app-backend-j25rgqsibtmlo.azurewebsites.net/chat or https://localhost:50505/chat The endpoint must respond with JSON.

[echen101]

I was using python 3.12, I tried 3.10 but there is the same error. I also noticed that promptflow was asking for keychain password during evaluate. I used the login password. After then the error occurred.

[pamelafox]

Ah! PromptFlow uses a keyring library that doesn't work as easily on a Mac. If you're working in the Dev Container, I added code to the Dockerfile to make the keyring work: https://github.com/Azure-Samples/ai-rag-chat-evaluator/blob/main/.devcontainer/Dockerfile

Here's the Promptflow FAQ about it: https://microsoft.github.io/promptflow/how-to-guides/faq.html#troubleshooting

I find that frustrating as well. Let me know if the tips in the FAQ work for you.

[jakebowles99]

I am also getting this issue when running on the devcontainer with the new keyring changes.

I'm trying to use the Azure OpenAI with key and not with user identity. They key is loaded into the .env var file as AZURE_OPENAI_KEY

Also tried on Ubuntu with the packages installed and same error

sudo apt-get install -y gcc cmake pkg-config libdbus-1-dev libglib2.0-dev
pip install keyrings.alt dbus-python

Error:

  File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/_sdk/_orm/connection.py", line 52, in get
    raise ConnectionNotFoundError(f"Connection {name!r} is not found.")
promptflow._sdk._errors.ConnectionNotFoundError: Connection 'Default_AzureOpenAI' is not found.

[pamelafox]

I haven't replicated this issue yet, but I am in discussion with the evaluate SDK team and PromptFlow team about the keyring related difficulties.

@jakebowles99 You said the "new keyring changes", was this working for you before a particular change?

Are both of you using keys or using default azure credential?

[echen101]

[update] I tested using api key as well, same issue. I am using default azure credential, not api keys.

[jakebowles99]

@jakebowles99 You said the "new keyring changes", was this working for you before a particular change?

@pamelafox Sorry I was referring to this, I had assumed it was a new change:

Ah! PromptFlow uses a keyring library that doesn't work as easily on a Mac. If you're working in the Dev Container, I added code to the Dockerfile to make the keyring work: https://github.com/Azure-Samples/ai-rag-chat-evaluator/blob/main/.devcontainer/Dockerfile

I have not yet had this working. I can generate the test prompts etc from the script, so it can access the OpenAI service and eval model, but the issue appears to lie in the promptflow side of things

[mhauschild]

@mhauschild That looks like a different issue, as its failing to get a JSON response from your chat endpoint. The target URL in example_config.json must correspond to the endpoint that receives POST requests from the chat app, like https://app-backend-j25rgqsibtmlo.azurewebsites.net/chat or https://localhost:50505/chat The endpoint must respond with JSON.

changed config to "target_url": "http://127.0.0.1:50505/chat", and getting now the same error as @jakebowles99 stated yesterday.

File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/_sdk/_orm/connection.py", line 52, in get raise ConnectionNotFoundError(f"Connection {name!r} is not found.") promptflow._sdk._errors.ConnectionNotFoundError: Connection 'Default_AzureOpenAI' is not found.

using api key.

my output:

File "C:\...\ai-rag-chat-evaluator\.venv\lib\site-packages\promptflow\_sdk\_orm\connection.py", line 52, in get raise ConnectionNotFoundError(f"Connection {name!r} is not found.") promptflow._sdk._errors.ConnectionNotFoundError: Connection 'Default_AzureOpenAI' is not found.

I'm on windows and python 3.10

[pamelafox]

Update: The issue is with the promptflow dependency, its latest release is incompatible. Please run:

pip install promptflow==1.6.0

[jakebowles99]

Thanks @pamelafox, works now.

[pamelafox]

Filed issue https://github.com/Azure/azure-sdk-for-python/issues/35060 for the azure-ai-generative team to investigate the incompatibility. Closing since it's working for us with the pinned version.

[echen101]

Using promptflow 1.6.0 fixes the issue. Thanks @pamelafox