Incorrect API key provided Azure OpenAI for TestsetGenerator

[llvll0hsen]

I am trying to use Azure Openai to create a test dataset. But for some reasons the generate_with_langchain_docs give the following error:

unable to apply transformation: Error code: 401 - {'error': {'message': 'Incorrect API key provided: 0ba2ec70********************407d. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}} when I test my api key using azure_llm.invoke("tell me a jok about weather") it work.

here is my code

import os
from ragas.testset import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

from langchain_openai.chat_models import AzureChatOpenAI
from langchain_openai.embeddings import AzureOpenAIEmbeddings
from langchain_community.document_loaders import WebBaseLoader

config = {
    "base_url": "https://xxxxx.openai.azure.com",
    "model_deployment": "my-gpt-4o-rag",
    "model_name": "gpt-4o",
    "embedding_deployment": "my-text-embedding-ada-002",
    "embedding_name": "text-embedding-ada-002",  }

loader = WebBaseLoader("https://www.advancinganalytics.co.uk/")
documents = loader.load()
for document in documents:
    document.metadata['filename'] = document.metadata['source']

azure_llm = AzureChatOpenAI(
    openai_api_version= "2024-08-01-preview",
    azure_endpoint=config["base_url"],
    azure_deployment=config["model_deployment"],
    model=config["model_name"],
    validate_base_url=False,
)

response = azure_llm.invoke("tell me a jok about the weather") #---> this works

azure_llm = LangchainLLMWrapper(azure_llm)
generator = TestsetGenerator.from_langchain(azure_llm)
testset = generator.generate_with_langchain_docs(
    documents,
    testset_size=5,
)

Ragas=0.2.1 and langchain_openai==0.2.2

[dosubot[bot]]

The error you're encountering suggests that the API key being used by the generate_with_langchain_docs method is incorrect or not being passed correctly. Here are a few things you can check and try:

1. API Key Configuration: Ensure that the API key is correctly set in your environment or configuration. The LangchainLLMWrapper and AzureChatOpenAI should be using the same API key. Double-check that the API key is correctly configured in your environment variables or wherever it is being set.

2. Environment Variables: If you're using environment variables to store your API key, make sure they are correctly set and accessible in the context where generate_with_langchain_docs is being called.

3. Wrapper Initialization: When you wrap AzureChatOpenAI with LangchainLLMWrapper, ensure that the wrapper is correctly passing the API key to the underlying model. There might be a discrepancy in how the wrapper handles the API key compared to direct invocation.

4. Version Compatibility: Although there are no known issues between ragas==0.2.1 and langchain_openai==0.2.2, ensure that all dependencies are up to date and compatible. Sometimes, updating to the latest versions can resolve unexpected issues.

5. Debugging: Add debugging logs to check if the API key is being altered or not passed correctly when using generate_with_langchain_docs. This can help identify where the issue might be occurring.

If these steps do not resolve the issue, consider isolating the problem by testing the API key with a minimal setup to ensure it works as expected in different parts of your code.

To continue talking to Dosu, mention @dosu.

[llvll0hsen]

I've already check the api key by calling the AzureChatOpenAI.invoke method and it works just fine.

[MariosKef]

@llvll0hsen I have exactly the same issue! It seems that the problem lies in the llm_factory. It falls back on ChatOpenAI, instead of AzureChatOpenAI. The docs here https://docs.ragas.io/en/stable/howtos/customizations/customize_models/#customize-models give an overview, but it still fails. @jjmachan can we expect a hotfix?

[Kefan-pauline]

Same here. The LLM on Azure works by itself, but not when it is embedded in TestsetGenerator, instead of using Azure API key, it uses OpenAI API key.

[jjmachan]

hey let me look into this at this ASAP and I will get back

[Kefan-pauline]

hey let me look into this at this ASAP and I will get back

Thanks! BTW, it is the same for embedding_factory. It would be nice to pass the llm and embedding model through the TestsetGenerator parameter and actually use them in the default transforms.

[hunter-walden2113]

I was having the same error. It seems like the llm_factory defaults to OpenAI and there's no support in that function for AzureOpenAI. I altered the code locally to try to see if it would accept an AzureOpenAI client but no luck. In the previous version (0.1.21) you used to be able to pass your LLM and embedding client directly to TestsetGenerator, but that seems to have changed in the latest version.

[MariosKef]

Are there any updates on this?

[hunter-walden2113]

I figured out the problem. I implemented a temporary fix locally just to have the ability to produce a test set, but it is not a long-term solution. I am willing to spend some time working on a fix if nobody else is.