"Error: OpenAI API key must be set for similarity comparison" comes up after using AZURE_OPENAI_API_KEY

[anoura12]

I'm using the Azure integration to run a similarity check for my prompts but this error comes up which asks me for my OPENAI_API_KEY instead. I've tried the overrides as well but that doesn't seem to work either.

I exported the AZURE_OPENAI_API_KEY separately in the command line.

promptfooconfig.yaml

prompts:
  - "Generate one very interesting fact about {{topic}}"

providers:
    - id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'llm.openai.azure.com'

tests:
  - vars:
      topic: bananas
    assert:
    - type: similar
      value: Bananas are naturally radioactive.
      provider: azureopenai:chat:gpt-35-turbo-16k

[typpo]

Hi @anoura12,

There might be something wrong with your setup as I am able to override the assertion provider with this config. Maybe you need to set the api host for the provider:

prompts:
  - "Generate one very interesting fact about {{topic}}"

providers:
    - id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'xxxx.openai.azure.com'

tests:
  - vars:
      topic: bananas
    assert:
      - type: similar
        value: Bananas are naturally radioactive.
        provider:
          id: azureopenai:chat:gpt-35-turbo-16k
          config:
            apiHost: 'xxxx.openai.azure.com'

Let me know if you're able to reproduce the issue with a minimal example.

[anoura12]

Hi @typpo

I tried adding the apiHost and the apiKey in the provider this time and it seems that I'm getting the same error again.

prompts:
  - "Generate one very interesting fact about {{topic}}"

providers:
    - id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'xxxx.openai.azure.com'
        apiKey: abc123

tests:
  - vars:
      topic: bananas
    assert:
      - type: similar
        value: Bananas are naturally radioactive.
        provider:
          id: azureopenai:chat:gpt-35-turbo-16k
          config:
            apiHost: 'xxxx.openai.azure.com'
            apiKey: abc123

Here is the error

Is there something I'm missing? Any idea what I should look into the Azure setup?

[typpo]

Ah I see, the similar assertion requires an embedding API and you are passing a chat API.

You'll have to create a deployment that uses the text-embedding-ada-002 model and give it azureopenai:embeddings:text-embedding-ada-002-deploymentname as the assertion provider.

Please give that a try and let me know how it works!

[anoura12]

Oh wow, this worked like a charm

Thanks so much!

[typpo]

No problem. I've updated the documentation and improved the error message, and will think about other ways to make Azure setup smoother.