Azure LLM evaluation - Deployment Error

[meenusel]

Describe the bug Trying to use an Azure API Key to run a LLM evaluation using UpTrain. I received a 404 error message saying that the deployment is not found. However, there is no deployment name parameter when creating the settings for the Eval LLM.

To Reproduce I followed the steps in this documentation: https://docs.uptrain.ai/llms/azure Error message: ERROR | uptrain.operators.language.llm:async_process_payload:103 - Error when sending request to LLM API: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

[ryancole0]

I don't think this is a bug, just looks like a documentation update would be helpful. The Azure OpenAI term "deployment" usually translates to "model" in other frameworks. The following works for me:

settings = Settings(model = 'azure/{}'.format(deployment_name), 
                    azure_api_key=os.getenv('API-KEY'), 
                    azure_api_version=api_version, 
                    azure_api_base=endpoint_url)
eval_llm = EvalLLM(settings)

[terry07]

Hello there.

That library seems great, but I still face issues when I try to sync with my Azure credentials.

None of the relevant comments did help me.

Can you please check for any inconsistency?

{ "name": "ValueError", "message": "OpenAI API Key is invalid", "stack": "--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[5], line 5 1 settings = Settings(model = 'azure/{}'.format('gpt-4'), 2 azure_api_key=os.getenv('AZURE_OPENAI_API_KEY'), 3 azure_api_version=os.getenv('OPENAI_API_VERSION'), 4 azure_api_base=os.getenv('AZURE_ENDPOINT')) ----> 5 eval_llm = EvalLLM(settings)

File ~/miniconda3/envs/eval/lib/python3.9/site-packages/uptrain/framework/evalllm.py:101, in EvalLLM.init(self, settings, openai_api_key) 99 response = check_openai_api_key(self.settings.openai_api_key) 100 if not response: --> 101 raise ValueError(\"OpenAI API Key is invalid\") 103 self.executor = APIClientWithoutAuth(self.settings)

ValueError: OpenAI API Key is invalid" }

[cthompson-insight]

This is failing when you check for an OpenAI key. The check_openai_api_key only checks if the key is valid using openai.

def check_openai_api_key(api_key):

    import openai
    client = openai.OpenAI(api_key=api_key)
    try:
        client.models.list()
    except openai.AuthenticationError:
        return False
    else:
        return True

Needs to also use the AzureOpenAI class to do the check.

[terry07]

Great, I replaced that part with the corresponding part regarding AzureOpenAI check.

from openai import AzureOpenAI

client = AzureOpenAI(
        api_key = openai_api_key,
        api_version=openai_api_version,
        azure_endpoint=openai_azure_endpoint,
            )

[deekshagarg39]

@terry07 Could you please help in How you are passing client to EvalLLM class

[NiklasElsaesser]

@terry07 Could you please help in How you are passing client to EvalLLM class

I would also appreciate to know how to execute this code properly!

[HelloJowet]

I'm using the following workaround until this issue is fixed:

from uptrain import EvalLLM, Settings
from uptrain.framework.remote import APIClientWithoutAuth

class EvalLLMWorkaround(EvalLLM):
    def __init__(self, settings: Settings = None, openai_api_key: str = None) -> None:
        if (openai_api_key is None) and (settings is None):
            raise Exception('Please provide OpenAI API Key')

        if settings is None:
            self.settings = Settings(openai_api_key=openai_api_key)
        else:
            self.settings = settings

        self.executor = APIClientWithoutAuth(self.settings)

OPENAI_API_KEY = "sk-********************"  # Insert your OpenAI key here
eval_llm = EvalLLMWorkaround(openai_api_key=OPENAI_API_KEY)

or

from uptrain import EvalLLM, Settings
from uptrain.framework.remote import APIClientWithoutAuth

class EvalLLMWorkaround(EvalLLM):
    def __init__(self, settings: Settings) -> None:
        self.settings = settings
        self.executor = APIClientWithoutAuth(self.settings)

# ...

[MouliChintakunta-AI]

azure_endpoint= os.getenv("AZURE_OPENAI_API_BASE")
azure_api_version = os.getenv("OPENAI_API_VERSION")
azure_api_key = os.getenv("AZURE_OPENAI_API_KEY")

from uptrain import EvalLLM, Evals, CritiqueTone, Settings
settings = Settings(model ='azure/{}'.format('AIF_V2'), azure_api_key=azure_api_key, azure_api_version=azure_api_version, azure_api_base=azure_endpoint)
eval_llm = EvalLLM(settings)

above code works for me for AzureLLM integration with uptrain