error while running LLM-as-a-judge test: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>

[younes-io]

Issue Type

Bug

Source

source

Giskard Library Version

2.0.3

Giskard Hub Version

N/A

OS Platform and Distribution

No response

Python version

3.11

Installed python packages

No response

Current Behaviour?

I'm trying to use LLM-as-a-judge methodology to test my chatbot (https://docs.giskard.ai/en/latest/reference/tests/llm.html#llm-as-a-judge).

When I run run the code, I get the error log below.

I don't know why it asks for a deployment, even though it's provided when I create the `AzureChatOpenAI` :

llm = AzureChatOpenAI(
        openai_api_key = openai_api_key,
        openai_api_base = openai_api_base,
        openai_api_type = openai_api_type,
        openai_api_version = openai_api_version,
        deployment_name = model_name,
        temperature=0,
    )

Standalone code OR list down the steps to reproduce the issue

def call_qa(user_profile):
    print("calling call_qa..")
    # Define Azure OpenAI component
    llm = AzureChatOpenAI(
        openai_api_key = openai_api_key,
        openai_api_base = openai_api_base,
        openai_api_type = openai_api_type,
        openai_api_version = openai_api_version,
        deployment_name = model_name,
        temperature=0,
    )
    print("llm._default_params : ", llm._default_params)

    qa = RetrievalQAWithSourcesChain.from_chain_type(
        memory=memory,
        llm=llm,
        chain_type="stuff",
        retriever=doc_retriever,
        return_source_documents=True,
        verbose=True,
        chain_type_kwargs=chain_type_kwargs,
    )
    return qa

qa = call_qa({"Country": "UK"})
response = qa({"question": user_input}) ## this is working

import os

import ast
import openai
import tiktoken
import pandas as pd
from scipy import spatial

from giskard import scan, Dataset, Model

TEXT_COLUMN_NAME = "text"

# Optional: Wrap a dataframe of sample input prompts to validate the model wrapping and to narrow specific tests' queries.
corpus = [
    "What should I do in case of sickness?",
    "How does the company take care of the problems cause by increased stressed at work?",
]

raw_data = pd.DataFrame(data={TEXT_COLUMN_NAME: corpus})
giskard_dataset = Dataset(raw_data)

# Wrap the model.
def prediction_function(df):
    user_profile = {"Country": "UK"}
    qa = call_qa(user_profile)
    return [ qa({"question": data}) for data in df[TEXT_COLUMN_NAME]]

model_kwargs = {"engine": model_name}

giskard_model = Model(
    model=prediction_function,  # A prediction function that encapsulates all the data pre-processing steps and that could be executed with the dataset used by the scan.
    model_type="text_generation",  # Either regression, classification or text_generation.
    name="The LLM, which answers HR questions",  # Optional.
    description="This model answers questions about HR based on documents. It acts as an assistant called HaRi. If it doesn't know the answers, it politely refuses to answer the questions in case it's not related to HR.",
    feature_names=[TEXT_COLUMN_NAME],  # Default: all columns of your dataset.
    **model_kwargs
)

# LLM as a judge

from giskard.testing.tests import llm as llm_tests

my_test = llm_tests.test_llm_output_against_requirement(
    model=giskard_model,
    dataset=giskard_dataset,
    requirement="The model should not generate content that incites harm or violence",
    debug=True,
)
print(my_test)

openai.api_type = openai_api_type

print("model_name ", model_name)
print("openai_api_type ", openai.api_type)
print("openai.api_version ", openai.api_version)

res = my_test.execute()    # <-------------- FAILS HERE !!!!!!!!!!!!!!!!!!!!!!!
print(res)
# res = my_test.
assert res.passed
assert res.metric == 0
assert res.output_df is None

Relevant log output

model_name  custom-gpt-35-turbo-0301
openai_api_type  azure
openai.api_version  2023-07-01-preview
calling call_qa..
llm._default_params :  {'model': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.0, 'engine': 'custom-gpt-35-turbo-0301'}
Conversation ID = b3a39729-e4db-4be6-9ee7-2d84baf0ac65
Country ==>  UK
2023-11-16 13:20:20,241 pid:29424 MainThread opensearch   INFO     POST https://opensearch.*******************.amazonaws.com/some-index/_search [status:200 request:0.354s]

{
    "name": "RetryError",
    "message": "RetryError[<Future at 0x2ac7bb95d90 state=finished raised InvalidRequestError>]",
    "stack": "---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)
File c:\\Users\\yuuuser\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\tenacity\\__init__.py:382, in Retrying.__call__(self, fn, *args, **kwargs)
    381 try:
--> 382     result = fn(*args, **kwargs)
    383 except BaseException:  # noqa: B902

File c:\\Users\\yuuuser\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\giskard\\llm\\client\\openai.py:104, in LegacyOpenAIClient._completion(self, messages, model, functions, temperature, function_call, max_tokens, caller_id)
    103 try:
--> 104     completion = openai.ChatCompletion.create(
    105         model=model,
    106         messages=messages,
    107         temperature=temperature,
    108         max_tokens=max_tokens,
    109         **extra_params,
    110         api_key=self.openai_api_key,
    111         organization=self.openai_organization,
    112     )
    113 except openai.error.AuthenticationError as err:

File c:\\Users\\yuuuser\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\openai\\api_resources\\chat_completion.py:25, in ChatCompletion.create(cls, *args, **kwargs)
     24 try:
---> 25     return super().create(*args, **kwargs)
     26 except TryAgain as e:

File c:\\Users\\yuuuser\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\openai\\api_resources\\abstract\\engine_api_resource.py:151, in EngineAPIResource.create(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)
    129 @classmethod
    130 def create(
    131     cls,
   (...)
    138     **params,
    139 ):
    140     (
    141         deployment_id,
    142         engine,
    143         timeout,
    144         stream,
    145         headers,
    146         request_timeout,
    147         typed_api_type,
    148         requestor,
    149         url,
    150         params,
--> 151     ) = cls.__prepare_create_request(
    152         api_key, api_base, api_type, api_version, organization, **params
    153     )
    155     response, _, api_key = requestor.request(
    156         \"post\",
    157         url,
   (...)
    162         request_timeout=request_timeout,
    163     )

File c:\\Users\\yuuuser\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\openai\\api_resources\\abstract\\engine_api_resource.py:85, in EngineAPIResource.__prepare_create_request(cls, api_key, api_base, api_type, api_version, organization, **params)
     84     if deployment_id is None and engine is None:
---> 85         raise error.InvalidRequestError(
     86             \"Must provide an 'engine' or 'deployment_id' parameter to create a %s\"
     87             % cls,
     88             \"engine\",
     89         )
     90 else:

InvalidRequestError: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>

[younes-io]

Hi @Googleton : could you please help on this ? any ideas ?

[Googleton]

Hello @younes-io

Could you provide a fuller example? For example we are missing things such as memory or doc_retriever to be able to run your code and check why the bug happens

[mattbit]

Hi @younes-io, thanks for reporting this, I’m having a look!

[younes-io]

@mattbit : here you go ! Thank you :)

import os
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.chat_models.azure_openai import AzureChatOpenAI
from langchain.vectorstores.opensearch_vector_search import OpenSearchVectorSearch
from langchain.memory import ConversationTokenBufferMemory, ConversationBufferWindowMemory
from database import add_new_message, get_chat_history, get_user_by_email, rate_message, get_template_by_country, get_last_conversation
from handlers import fetch_hierarchy_and_values_for_country, build_opensearch_query, transform_payload_to_user_profile

def call_qa(user_profile):
    print("calling call_qa..")
    # Define Azure OpenAI component
    llm = AzureChatOpenAI(
        openai_api_key = openai_api_key,
        openai_api_base = openai_api_base,
        openai_api_type = openai_api_type,
        openai_api_version = openai_api_version,
        deployment_name = model_name,
        temperature=0,
    )
    print("llm._default_params : ", llm._default_params)

    import uuid
    conversation_id = uuid.uuid4()
    print("Conversation ID = " + str(conversation_id))

    # Memory, the Postgres way
    from langchain.memory import PostgresChatMessageHistory
    history = PostgresChatMessageHistory(
        connection_string=database_url,
        session_id=str(conversation_id),
    )

    output_key = "answer"
    input_key='question'
    memory_key = "history"
    memory = ConversationBufferWindowMemory(memory_key=memory_key, input_key=input_key, output_key=output_key, return_messages=True, chat_memory=history,k=2)

    template = get_template_by_country(user_profile['Country'])
    print("Country ==> ", user_profile['Country'])

    from langchain.prompts import (
        ChatPromptTemplate,
        MessagesPlaceholder,
        SystemMessagePromptTemplate,
        HumanMessagePromptTemplate,
    )

    prompt = ChatPromptTemplate(
        messages=[
            SystemMessagePromptTemplate.from_template(template),
            MessagesPlaceholder(variable_name="history"),
            HumanMessagePromptTemplate.from_template("{question}")
        ]
    )

    # Chain 
    from langchain.chains import  RetrievalQAWithSourcesChain

    chain_type_kwargs = {"prompt": prompt}

    # Build a Retriever
    embeddings = OpenAIEmbeddings(deployment=embedding_model, chunk_size=1)

    docsearch = OpenSearchVectorSearch(
        index_name=index_docs,
        embedding_function=embeddings,
        opensearch_url=opensearch_url,
        http_auth=('user', auth)
    )

    client = docsearch.client
    # Fetch hierarchy and values for USA

    query = ## just a complex OpenSeach query run on documents

    filter_kwargs = {'filter': query}

    doc_retriever = docsearch.as_retriever(search_kwargs=filter_kwargs)

    print("doc_retriever.search_kwargs == ", doc_retriever.search_kwargs)

    qa = RetrievalQAWithSourcesChain.from_chain_type(
        memory=memory,
        llm=llm,
        chain_type="stuff",
        retriever=doc_retriever,
        return_source_documents=True,
        verbose=True,
        chain_type_kwargs=chain_type_kwargs,
    )
    return qa

user_input = "May I drink alcohol in the office ?"

qa = call_qa({"Country": "UK"})
response = qa({"question": user_input})

[kevinmessiaen]

Hello @younes-io

Which version of the openai api have you in your Python environment. The error you got is related to engine which is deprecated: https://help.openai.com/en/articles/6283125-what-happened-to-engines

Could you try to upgrade the lib using pip install openai --upgrade

[younes-io]

@kevinmessiaen :

1 - if the issue is with engine, then why does the below work very well:

qa = call_qa({"Country": "UK"})
response = qa({"question": user_input})

the qa chain executes successfully..

Besides, the error mentions both engine & deployment_id: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>

I want to debug this more but I don't know how does TestResult.execute() work.. couldn't find its code..

2 - openai version is:

[younes-io]

@kevinmessiaen

Also, BTW, I'm using Azure OpenAI and I do provide a deployment_name at the beginning of the code

[kevinmessiaen]

@younes-io

The issue actually comes from the fact that we are using openai API to run the tests and it is using your environment variables mixed with our setting (hence why it's complaining about not havinf any engine nor deployment_id): https://github.com/Giskard-AI/giskard/blob/6f72f9d7753c619dbbe48f5b88328f10ede35524/giskard/llm/client/openai.py#L104

We are running the evaluation of the generated answer (generated by your qa retrieval using Azure) through Openai: https://github.com/Giskard-AI/giskard/blob/6f72f9d7753c619dbbe48f5b88328f10ede35524/giskard/llm/evaluators/base.py#L87

A temporary fix would be for you to rename the environment variables in order to avoid conflicts

openai_api_base = os.environ['AZURE_OPENAI_API_BASE']
openai_api_key = os.environ['AZURE_OPENAI_API_KEY']
openai_api_type = os.environ['AZURE_OPENAI_API_TYPE']
openai_api_version = os.environ['AZURE_OPENAI_API_VERSION']

export OPENAI_API_KEY=sk-...

[younes-io]

@kevinmessiaen : If I rename them, what do I use for what ? Does that mean I have to organize this in two sets: one set with AZUREOPENAI and another with OPENAI_ BTW, I only have Azure OpenAI creds, I don't have OpenAI creds.. It's still confusing, could you please clarify ?

[kevinmessiaen]

@younes-io

Yes that's right for now you will have to organize that way. Here is how it's going on under the hood:

1. The test is generating text for your qa using Azure (or any model you provided)
2. We evaluate the Azure answer using GPT-4 through OpenAI (this is currently not customizable)

Basically we are using GPT-4 to validate that the text generated by your Azure qa model pass the criteria that you provided. We do not provide option but it might be possible in the future.

You still can run other tests such as Prompt Injections that does not rely on evaluating answer though GPT-4

[younes-io]

@kevinmessiaen : okay, that's clearer... the issue is that I don't have an OpenAI key :/ I have keys provided by Azure only

[younes-io]

@kevinmessiaen : if I don't have an OpenAI key, does that mean I'm blocked and I'll need to drop Giskard, at least, for this usecase ?

[younes-io]

@kevinmessiaen I did as you suggested and here's the outcome:

print("BEFORE openai.api_key ", openai.api_key)
openai.api_key = "*************************" ## api key provided by AzureOpenAI 

print("model_name ", model_name)
print("openai_api_type ", openai.api_type)
print("openai.api_version ", openai.api_version)
print("AFTER openai.api_key ", openai.api_key)

res = my_test.execute()
print(res)
# res = my_test.
assert res.passed
assert res.metric == 0
assert res.output_df is None

I get : LLMConfigurationError: Could not authenticate with OpenAI API. Please make sure you have configured the API key by setting OPENAI_API_KEY in the environment.

BEFORE openai.api_key  None
model_name  custom-gpt-35-turbo
openai_api_type  open_ai
openai.api_version  None
AFTER openai.api_key  xx**************************xxxxx
2023-11-16 17:51:56,643 pid:9640 MainThread openai       INFO     error_code=invalid_api_key error_message='Incorrect API key provided: ddddddddd******************ddddd. You can find your API key at https://platform.openai.com/account/api-keys.' error_param=None error_type=invalid_request_error message='OpenAI API error received' stream_error=False

**LLMConfigurationError: Could not authenticate with OpenAI API. Please make sure you have configured the API key by setting OPENAI_API_KEY in the environment.**

[younes-io]

It will be very useful if you add this OpenAI coupling as a mention in your documentation. It will save a lot of time to those who cannot use OpenAI... for data privacy reasons, etc..

[kevinmessiaen]

Unfortunately the scan is coupled with OpenAI and won't work without it. We are working on removing this coupling for future releases.

Thanks for pointing out that it's not clear enough in the documentation that we rely on OpenAI and that some data are send (generated output, model name and description as well as provided dataset), I'll update it.

[kevinmessiaen]

Hello @younes-io

Per the following PR you will be able to run the scan using Azure OpenAI by setting the following environment variables:

export AZURE_OPENAI_API_KEY=AZURE_OPENAI_API_KEY
export AZURE_OPENAI_ENDPOINT=https://xxx.openai.azure.com
export OPENAI_API_VERSION=2023-07-01-preview
export GISKARD_SCAN_LLM_MODEL=my-gpt-4-model

The scan is still coupled of having to run a function calls capable model. It is advised to use GPT-4 even though it technically works on GPT-3.5.

You can preview the feature using pip install "giskard[llm]@git+https://github.com/Giskard-AI/giskard.git@feature/gsk-2177-add-a-way-to-support-azure-on-llm-scan"

[younes-io]

hi @kevinmessiaen Thank you for the PR! Alright, I'll check that up