Script provider is asking me to supply OPEN_AI_KEY

[geekyme-fsmk]

Getting this error when I run the evaluation:

[FAIL] Error: OpenAI API key is not set. Set OPENAI_API_KEY environment variable or pass it as an argument to the constructor. Error: OpenAI API key is not set. Set OPENAI_API_KEY environment variable or pass it as an argument to the constructor.
--

https://www.promptfoo.dev/docs/configuration/testing-llm-chains#using-a-script-provider

Following the guide above, I set up a custom script provider:

prompts:
  - tests/prompt/prompt1.txt
providers:
  - exec:poetry run python tests/prompt/ai_client_provider.py
tests:
  - vars:
      question: What's the weather in New York?
    assert:
      - type: llm-rubric
        value: does not claim to know the current weather in New York

How the ai_client_provider.py looks like:

import sys
from app.clients.ai_client import AIClient
from app.services.intent_service import IntentService

conversation_id = "dummy"
admin_id = "dummy"
ai_client = AIClient()
intent = IntentService(conversation_id, admin_id, ai_client.vectorstores)

chat_history = []
question = sys.argv[1]
response_text = ai_client.get_response(question, chat_history, intent)

print(response_text)

AIClient internally would load its own azure openai keys and secrets. Not sure why I am getting the errors from here when I've already specified a custom script.

My config does not appear to be going down the script completion path.

promptfoo --version => 0.20.1

[typpo]

Hi @geekyme-fsmk,

The llm-rubric assert type uses GPT-4 by default, which is why it's asking for the OpenAI API key. You can override the LLM grader using one of these methods: https://promptfoo.dev/docs/configuration/expected-outputs#overriding-the-llm-grader

Note that the grader uses a chat-completion style prompt by default. If you need to override it or modify the prompt, you can set the rubricPrompt property on the test or assertion: https://promptfoo.dev/docs/configuration/expected-outputs#assertion-properties

Hope this helps solve your issue!

[geekyme-fsmk]

hi @typpo , understood - is it possible then to use my Azure Open AI key instead? I do have a subscription already with GPT4 enabled.

[geekyme-fsmk]

Alright I ended up doing this:

defaultTest:
  options:
    provider: azureopenai:chat:gpt-35-turbo

Then specifying my env vars accordingly:

AZURE_OPENAI_API_HOST=<azure_base> AZURE_OPENAI_API_KEY=<api_key> promptfoo eval -c promptfooconfig.yaml