search

explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.31k stars 746 forks source link

TypeError: create() got an unexpected keyword argument 'deployment_name' #470

Closed abhijithneilabraham closed 5 months ago

abhijithneilabraham commented 10 months ago

TypeError: create() got an unexpected keyword argument 'deployment_name'

I encountered this issue while trying to run this example

Ragas version: ragas==0.0.22 Python version: 3.9

Code to Reproduce

from datasets import load_dataset
import os
from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
)
from ragas.metrics.critique import harmfulness
from langchain.chat_models import AzureChatOpenAI
from langchain.embeddings import AzureOpenAIEmbeddings
from ragas.llms import LangchainLLM
from ragas import evaluate

# list of metrics we're going to use
metrics = [
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
    harmfulness,
]
fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")
print(fiqa_eval)

# Set the OpenAI API key
os.environ["AZURE_OPENAI_API_KEY"] = "******"
os.environ["OPENAI_API_KEY"] = "******"
# os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_VERSION"] ="******"

azure_model = AzureChatOpenAI(
    deployment_name="******",
    model="gpt-35-turbo",
    openai_api_base="******",
    openai_api_type="azure",
)
# wrapper around azure_model
ragas_azure_model = LangchainLLM(azure_model)
answer_relevancy.llm = ragas_azure_model

# init and change the embeddings
# only for answer_relevancy
azure_embeddings = AzureOpenAIEmbeddings(
    deployment_name="******",
    model="gpt-35-turbo",
    openai_api_base="******",
    openai_api_type="azure",
)

# embeddings can be used as it is
answer_relevancy.embeddings = azure_embeddings

for m in metrics:
    m.__setattr__("llm", ragas_azure_model)

fiqa_eval = load_dataset("explodinggradients/fiqa", "ragas_eval")
result = evaluate(
    fiqa_eval["baseline"],
    metrics=metrics,
)

print(result)

Error trace

/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import AzureOpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import OpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import AzureChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import BedrockChat`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatVertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import AmazonAPIGateway`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import AzureOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import Bedrock`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import OpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import VertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
DatasetDict({
    baseline: Dataset({
        features: ['question', 'ground_truths', 'answer', 'contexts'],
        num_rows: 30
    })
})
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.chat_models.azure_openai.AzureChatOpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import AzureChatOpenAI`.
  warn_deprecated(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/chat_models/azure_openai.py:174: UserWarning: As of openai>=1.0.0, if `deployment_name` (or alias `azure_deployment`) is specified then `openai_api_base` (or alias `base_url`) should not be. Instead use `deployment_name` (or alias `azure_deployment`) and `azure_endpoint`.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/chat_models/azure_openai.py:182: UserWarning: As of openai>=1.0.0, if `openai_api_base` (or alias `base_url`) is specified it is expected to be of the form https://example-resource.azure.openai.com/openai/deployments/example-deployment. Updating https://mvpgomi.openai.azure.com/openai to https://mvpgomi.openai.azure.com/openai.
  warnings.warn(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.embeddings.azure_openai.AzureOpenAIEmbeddings` was deprecated in langchain-community 0.1.0 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import AzureOpenAIEmbeddings`.
  warn_deprecated(
/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/embeddings/openai.py:268: UserWarning: WARNING! deployment_name is not default parameter.
                    deployment_name was transferred to model_kwargs.
                    Please confirm that deployment_name is what you intended.
  warnings.warn(
evaluating with [faithfulness]
100%|█████████████████████████████████████████████| 2/2 [02:41<00:00, 80.59s/it]
evaluating with [answer_relevancy]
  0%|                                                     | 0/2 [00:16<?, ?it/s]
Traceback (most recent call last):
  File "/Users/abhijithneilabraham/Desktop/ragas_test.py", line 62, in <module>
    result = evaluate(
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/ragas/evaluation.py", line 101, in evaluate
    scores.append(metric.score(dataset).select_columns(metric.name))
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/ragas/metrics/base.py", line 76, in score
    score = self._score_batch(dataset.select(batch), callbacks=group)
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/ragas/metrics/_answer_relevance.py", line 136, in _score_batch
    cosine_sim = self.calculate_similarity(question, gen_questions)
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/ragas/metrics/_answer_relevance.py", line 145, in calculate_similarity
    question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1)
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/embeddings/openai.py", line 697, in embed_query
    return self.embed_documents([text])[0]
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/embeddings/openai.py", line 668, in embed_documents
    return self._get_len_safe_embeddings(texts, engine=engine)
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/embeddings/openai.py", line 494, in _get_len_safe_embeddings
    response = embed_with_retry(
  File "/Users/abhijithneilabraham/mambaforge/envs/py/lib/python3.9/site-packages/langchain_community/embeddings/openai.py", line 116, in embed_with_retry
    return embeddings.client.create(**kwargs)
TypeError: create() got an unexpected keyword argument 'deployment_name'

I also tried replacing the langchain imports this way but the issue still persists:

from langchain_community.chat_models import AzureChatOpenAI
from langchain_community.embeddings import AzureOpenAIEmbeddings

Kindly let me know if I am missing some detail here.

shahules786 commented 10 months ago

Hi @abhijithneilabraham , if you're using AzureOpenAIEmbeddings can you try changing arguments according to doc here from langchain